2024-04-22T06:07:24.422 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T06:07:24.422 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179 --verbose --name yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi -- /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/orig.config.yaml 2024-04-22T06:07:24.483 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:07:24.564 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179 branch: ci-yingxin-crimson-fix-tri-mutex-4-base description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7668179' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-fix-tri-mutex-4-base ceph: conf: client.0: admin socket: /var/run/ceph/ceph-$name.asok global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: ci-yingxin-crimson-fix-tri-mutex-4-base env: CRIMSON_COMPAT: '1' sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-fix-tri-mutex-4-base suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCArwe8JY69vRPI1gOppoAfuoqH5tvZqOUAgJncBEpBp/SABpASB1UDcRygqcH3J18xC9OFcHZCwRO4oq2Y24ak= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGX9O9cuJkHoW43qGAdaoeyYRTcsLwDfmJ7iTXZTE/RP482xLx50nETZMnXxLAlwMqOuk7JBZyCUl9Qk+TYaugA= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - radosbench: clients: - client.0 time: 150 - admin_socket: client.0: objecter_requests: test: http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch} teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_05:57:37 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T06:07:24.565 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa; will attempt to use it 2024-04-22T06:07:24.565 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks 2024-04-22T06:07:24.565 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T06:07:24.567 INFO:teuthology.task.internal:Checking packages... 2024-04-22T06:07:24.589 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '8d82defa854b9197ed2c2b855ac5e996c480ec88' 2024-04-22T06:07:24.589 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T06:07:24.589 INFO:teuthology.packaging:ref: None 2024-04-22T06:07:24.589 INFO:teuthology.packaging:tag: None 2024-04-22T06:07:24.589 INFO:teuthology.packaging:branch: ci-yingxin-crimson-fix-tri-mutex-4-base 2024-04-22T06:07:24.589 INFO:teuthology.packaging:sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:07:24.590 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-fix-tri-mutex-4-base 2024-04-22T06:07:24.805 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3189.g8d82defa 2024-04-22T06:07:24.806 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T06:07:24.815 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T06:07:24.815 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T06:07:24.822 INFO:teuthology.task.internal:Saving configuration 2024-04-22T06:07:24.836 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T06:07:24.842 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:07:24.859 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:02:01.188758', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCArwe8JY69vRPI1gOppoAfuoqH5tvZqOUAgJncBEpBp/SABpASB1UDcRygqcH3J18xC9OFcHZCwRO4oq2Y24ak='} 2024-04-22T06:07:24.875 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:02:01.187508', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGX9O9cuJkHoW43qGAdaoeyYRTcsLwDfmJ7iTXZTE/RP482xLx50nETZMnXxLAlwMqOuk7JBZyCUl9Qk+TYaugA='} 2024-04-22T06:07:24.875 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T06:07:24.881 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-22T06:07:24.881 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-22T06:07:24.881 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T06:07:24.951 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f988ea5be50>, signals=[15]) 2024-04-22T06:07:24.951 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T06:07:24.957 INFO:teuthology.task.internal:Opening connections... 2024-04-22T06:07:24.957 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T06:07:24.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:07:25.044 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T06:07:25.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:07:25.118 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T06:07:25.126 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-22T06:07:25.144 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-22T06:07:25.144 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:07:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:07:25.204 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:07:25.204 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-22T06:07:25.222 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-22T06:07:25.240 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-22T06:07:25.240 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:07:25.298 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:07:25.299 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:07:25.299 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:07:25.299 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:07:25.299 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:07:25.299 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-22T06:07:25.314 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T06:07:25.323 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T06:07:25.328 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T06:07:25.329 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:07:25.331 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:07:25.354 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T06:07:25.360 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T06:07:25.360 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:07:25.388 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:07:25.515 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T06:07:25.558 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T06:07:25.573 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-22T06:07:25.573 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-22T06:07:25.573 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:07:25.574 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:07:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:07:25.590 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-22T06:07:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:07:25.591 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-22T06:07:25.647 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:07:25.648 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-437.el9.x86_64 2024-04-22T06:07:25.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-04-22T06:07:25.649 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:07:25.649 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-437.el9.x86_64 2024-04-22T06:07:25.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-04-22T06:07:27.908 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T06:07:30.092 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-04-22T06:07:31.893 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-22T06:07:37.127 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-22T06:07:39.229 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-22T06:07:39.963 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 34 MB/s | 21 MB 00:00 2024-04-22T06:07:44.979 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-22T06:07:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:07:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:07:46.958 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:07:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:07:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:07:47.075 DEBUG:teuthology.orchestra.run.smithi103:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:07:47.570 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:07:44 AM UTC. 2024-04-22T06:07:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Reinstalling: 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 255 k 2024-04-22T06:07:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 0 2024-04-22T06:07:47.713 INFO:teuthology.orchestra.run.smithi103.stderr:Operation aborted. 2024-04-22T06:07:47.774 INFO:teuthology.orchestra.run.smithi103.stdout:Is this ok [y/N]: 2024-04-22T06:07:47.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum reinstall -y kernel || true 2024-04-22T06:07:48.265 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 06:07:44 AM UTC. 2024-04-22T06:07:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:07:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:07:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-22T06:07:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:07:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:Reinstalling: 2024-04-22T06:07:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 255 k 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 0 2024-04-22T06:07:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:07:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-437.el9.x86_64.rpm 575 kB/s | 255 kB 00:00 2024-04-22T06:07:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:07:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:Total 355 kB/s | 255 kB 00:00 2024-04-22T06:07:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:07:49.301 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:07:49.301 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:07:49.306 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:07:49.306 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:07:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:07:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:07:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:07:50.138 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 922 kB/s | 19 MB 00:21 2024-04-22T06:07:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:07:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:07:50.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:07:50.994 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:07:50.995 INFO:teuthology.orchestra.run.smithi103.stdout:Reinstalled: 2024-04-22T06:07:50.995 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:07:50.995 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:07:50.995 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:07:51.176 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:07:51.223 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:07:51.223 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:07:51.223 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:07:51.223 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-22T06:07:51.224 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T06:07:51.251 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-04-22T06:07:51.272 DEBUG:teuthology.parallel:result is None 2024-04-22T06:07:55.697 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-04-22T06:07:57.364 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-22T06:08:00.068 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-04-22T06:08:05.249 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-04-22T06:08:07.138 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:08:07.139 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:08:07.229 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:08:07.230 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:08:07.230 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:08:07.341 DEBUG:teuthology.orchestra.run.smithi028:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:08:07.903 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 06:08:05 AM UTC. 2024-04-22T06:08:08.042 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-22T06:08:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:Operation aborted. 2024-04-22T06:08:08.112 INFO:teuthology.orchestra.run.smithi028.stdout:Is this ok [y/N]: 2024-04-22T06:08:08.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum reinstall -y kernel || true 2024-04-22T06:08:08.629 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:08:05 AM UTC. 2024-04-22T06:08:08.750 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-22T06:08:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:08:09.051 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T06:08:09.051 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:08:09.051 INFO:teuthology.orchestra.run.smithi028.stdout:Total 848 kB/s | 255 kB 00:00 2024-04-22T06:08:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:08:09.262 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:08:09.262 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:08:09.267 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:08:09.268 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:08:09.417 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:08:09.477 INFO:teuthology.orchestra.run.smithi028.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:08:09.477 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:08:10.123 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:08:10.123 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:08:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:08:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:08:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalled: 2024-04-22T06:08:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:08:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:08:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:08:11.022 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:08:11.070 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:08:11.070 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:08:11.070 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:08:11.070 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T06:08:11.070 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T06:08:11.103 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-04-22T06:08:11.121 DEBUG:teuthology.parallel:result is None 2024-04-22T06:08:11.121 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T06:08:11.129 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T06:08:11.129 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:08:11.164 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:08:11.182 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T06:08:11.189 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T06:08:11.198 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T06:08:11.198 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:08:11.219 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:08:11.272 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T06:08:11.279 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T06:08:11.279 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:08:11.282 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:08:11.315 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:08:11.324 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:08:11.342 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:08:11.352 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:08:11.354 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T06:08:11.361 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T06:08:11.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:08:11.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:08:11.394 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T06:08:11.403 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T06:08:11.403 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:08:11.434 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:08:11.450 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:08:11.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:08:11.572 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:08:11.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:08:11.693 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:08:11.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:08:11.757 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:08:11.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:08:11.849 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:08:11.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:08:11.970 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:08:11.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:08:12.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-22T06:08:12.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-22T06:08:12.073 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:08:12.122 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:08:12.503 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T06:08:12.511 INFO:teuthology.task.internal:Starting timer... 2024-04-22T06:08:12.511 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T06:08:12.522 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T06:08:12.531 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-22T06:08:12.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-22T06:08:12.583 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-22T06:08:12.585 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-22T06:08:12.626 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-22T06:08:12.628 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T06:08:12.628 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-22T06:08:12.651 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-22T06:08:12.651 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-22T06:08:12.695 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-22T06:08:12.695 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-04-22T06:08:12.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:08:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:08:12.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:08:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:08:12.760 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T06:08:12.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:08:12.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:08:12.824 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T06:08:12.834 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T06:08:12.851 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T06:08:12.853 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T06:11:44.703 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-04-22T06:11:44.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-22T06:11:44.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:11:44.783 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-22T06:11:44.865 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-22T06:11:44.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-22T06:11:44.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:11:44.944 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-22T06:11:45.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-22T06:11:45.025 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T06:11:45.036 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T06:11:45.037 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:11:45.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:11:45.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:11:45.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:11:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:11:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:11:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:11:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-22T06:11:45.108 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:11:45.115 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-22T06:11:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:11:45.132 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-22T06:11:45.142 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-22T06:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:11:45.156 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:11:45.170 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:11:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:11:45.213 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:11:45.213 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-22T06:11:45.213 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.213 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.213 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.213 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.231 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:11:45.234 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:11:45.234 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-22T06:11:45.234 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.234 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.234 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.234 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:11:45.235 INFO:teuthology.run_tasks:Running task install... 2024-04-22T06:11:45.244 DEBUG:teuthology.task.install:project ceph 2024-04-22T06:11:45.244 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}} 2024-04-22T06:11:45.244 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'} 2024-04-22T06:11:45.244 INFO:teuthology.task.install:Using flavor: crimson 2024-04-22T06:11:45.250 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T06:11:45.250 INFO:teuthology.task.install:extra packages: [] 2024-04-22T06:11:45.250 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'tag': None, 'wait_for_package': False} 2024-04-22T06:11:45.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:11:45.253 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'tag': None, 'wait_for_package': False} 2024-04-22T06:11:45.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:11:45.395 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/ 2024-04-22T06:11:45.395 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3189.g8d82defa 2024-04-22T06:11:45.473 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/ 2024-04-22T06:11:45.473 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3189.g8d82defa 2024-04-22T06:11:45.568 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:11:45.568 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:11:45.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:11:45.593 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T06:11:45.593 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8d82defa854b9197ed2c2b855ac5e996c480ec88/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:11:45.648 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:11:45.648 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:11:45.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:11:45.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:11:45.675 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T06:11:45.675 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8d82defa854b9197ed2c2b855ac5e996c480ec88/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:11:45.731 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:11:45.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:11:45.793 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-22T06:11:45.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-22T06:11:45.814 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:11:45.878 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-22T06:11:45.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-22T06:11:46.024 INFO:teuthology.orchestra.run.smithi103.stdout:48 files removed 2024-04-22T06:11:46.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-04-22T06:11:46.113 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-22T06:11:46.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw 2024-04-22T06:11:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 217 kB/s | 87 kB 00:00 2024-04-22T06:11:46.728 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 275 kB/s | 87 kB 00:00 2024-04-22T06:11:47.095 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-22T06:11:47.159 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-22T06:11:47.471 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 6.0 kB/s | 2.0 kB 00:00 2024-04-22T06:11:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 5.4 kB/s | 2.0 kB 00:00 2024-04-22T06:11:48.012 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-22T06:11:48.351 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T06:11:50.022 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-22T06:11:50.088 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-22T06:11:54.281 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-22T06:11:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-04-22T06:11:55.983 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-22T06:11:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-22T06:12:15.068 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-22T06:12:16.218 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-22T06:12:20.249 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-22T06:12:21.314 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-22T06:12:22.519 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:12:22.521 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:12:22.521 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:12:22.521 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:12:22.521 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:12:22.521 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 10 M 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.7 M 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.9 M 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 19 M 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 25 k 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:12:22.522 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 796 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 547 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.3 M 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 45 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 135 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 159 k 2024-04-22T06:12:22.523 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 315 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 293 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 99 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout:Install 30 Packages 2024-04-22T06:12:22.524 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-04-22T06:12:22.525 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:22.525 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 59 M 2024-04-22T06:12:22.525 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:12:23.483 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:12:23.486 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:12:23.486 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:12:23.486 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:12:23.486 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:12:23.486 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 10 M 2024-04-22T06:12:23.487 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-22T06:12:23.487 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:12:23.487 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.7 M 2024-04-22T06:12:23.487 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:12:23.487 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:12:23.487 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.9 M 2024-04-22T06:12:23.488 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 19 M 2024-04-22T06:12:23.488 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 25 k 2024-04-22T06:12:23.488 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:12:23.488 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:12:23.488 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:12:23.488 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 796 k 2024-04-22T06:12:23.489 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:12:23.489 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:12:23.489 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:12:23.489 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 547 k 2024-04-22T06:12:23.489 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:12:23.489 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.3 M 2024-04-22T06:12:23.490 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:12:23.490 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:12:23.490 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:12:23.490 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 45 k 2024-04-22T06:12:23.490 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 135 k 2024-04-22T06:12:23.491 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 159 k 2024-04-22T06:12:23.491 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:12:23.491 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 315 k 2024-04-22T06:12:23.491 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 293 k 2024-04-22T06:12:23.491 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 99 k 2024-04-22T06:12:23.491 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:12:23.492 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:12:23.492 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:12:23.492 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:12:23.492 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-22T06:12:23.492 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:12:23.492 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:23.493 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:12:23.493 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:12:23.493 INFO:teuthology.orchestra.run.smithi103.stdout:Install 30 Packages 2024-04-22T06:12:23.493 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-04-22T06:12:23.493 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:23.494 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 59 M 2024-04-22T06:12:23.494 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:12:23.783 INFO:teuthology.orchestra.run.smithi028.stdout:(1/32): ceph-radosgw-19.0.0-3189.g8d82defa.el9. 9.8 MB/s | 10 MB 00:01 2024-04-22T06:12:23.850 INFO:teuthology.orchestra.run.smithi028.stdout:(2/32): ceph-selinux-19.0.0-3189.g8d82defa.el9. 379 kB/s | 25 kB 00:00 2024-04-22T06:12:23.926 INFO:teuthology.orchestra.run.smithi028.stdout:(3/32): ceph-base-19.0.0-3189.g8d82defa.el9.x86 5.1 MB/s | 5.9 MB 00:01 2024-04-22T06:12:23.959 INFO:teuthology.orchestra.run.smithi028.stdout:(4/32): libcephfs2-19.0.0-3189.g8d82defa.el9.x8 7.2 MB/s | 796 kB 00:00 2024-04-22T06:12:24.067 INFO:teuthology.orchestra.run.smithi028.stdout:(5/32): libradosstriper1-19.0.0-3189.g8d82defa. 3.8 MB/s | 547 kB 00:00 2024-04-22T06:12:24.185 INFO:teuthology.orchestra.run.smithi028.stdout:(6/32): python3-ceph-argparse-19.0.0-3189.g8d82 386 kB/s | 45 kB 00:00 2024-04-22T06:12:24.310 INFO:teuthology.orchestra.run.smithi028.stdout:(7/32): python3-ceph-common-19.0.0-3189.g8d82de 1.1 MB/s | 135 kB 00:00 2024-04-22T06:12:24.427 INFO:teuthology.orchestra.run.smithi028.stdout:(8/32): python3-cephfs-19.0.0-3189.g8d82defa.el 1.3 MB/s | 159 kB 00:00 2024-04-22T06:12:24.494 INFO:teuthology.orchestra.run.smithi028.stdout:(9/32): librgw2-19.0.0-3189.g8d82defa.el9.x86_6 9.8 MB/s | 5.3 MB 00:00 2024-04-22T06:12:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:(10/32): python3-rados-19.0.0-3189.g8d82defa.el 3.1 MB/s | 315 kB 00:00 2024-04-22T06:12:24.569 INFO:teuthology.orchestra.run.smithi028.stdout:(11/32): python3-rbd-19.0.0-3189.g8d82defa.el9. 3.8 MB/s | 293 kB 00:00 2024-04-22T06:12:24.619 INFO:teuthology.orchestra.run.smithi028.stdout:(12/32): python3-rgw-19.0.0-3189.g8d82defa.el9. 1.1 MB/s | 99 kB 00:00 2024-04-22T06:12:24.686 INFO:teuthology.orchestra.run.smithi028.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-04-22T06:12:24.712 INFO:teuthology.orchestra.run.smithi028.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 752 kB/s | 107 kB 00:00 2024-04-22T06:12:24.762 INFO:teuthology.orchestra.run.smithi028.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-04-22T06:12:24.846 INFO:teuthology.orchestra.run.smithi028.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-04-22T06:12:24.871 INFO:teuthology.orchestra.run.smithi028.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-22T06:12:24.921 INFO:teuthology.orchestra.run.smithi028.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-04-22T06:12:24.972 INFO:teuthology.orchestra.run.smithi028.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-22T06:12:24.997 INFO:teuthology.orchestra.run.smithi028.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 477 kB/s | 36 kB 00:00 2024-04-22T06:12:25.047 INFO:teuthology.orchestra.run.smithi028.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 631 kB/s | 47 kB 00:00 2024-04-22T06:12:25.147 INFO:teuthology.orchestra.run.smithi028.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2024-04-22T06:12:25.166 INFO:teuthology.orchestra.run.smithi103.stdout:(1/32): ceph-base-19.0.0-3189.g8d82defa.el9.x86 4.2 MB/s | 5.9 MB 00:01 2024-04-22T06:12:25.190 INFO:teuthology.orchestra.run.smithi028.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 617 kB/s | 26 kB 00:00 2024-04-22T06:12:25.232 INFO:teuthology.orchestra.run.smithi028.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-22T06:12:25.232 INFO:teuthology.orchestra.run.smithi103.stdout:(2/32): ceph-selinux-19.0.0-3189.g8d82defa.el9. 378 kB/s | 25 kB 00:00 2024-04-22T06:12:25.274 INFO:teuthology.orchestra.run.smithi028.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-22T06:12:25.374 INFO:teuthology.orchestra.run.smithi103.stdout:(3/32): libcephfs2-19.0.0-3189.g8d82defa.el9.x8 5.5 MB/s | 796 kB 00:00 2024-04-22T06:12:25.508 INFO:teuthology.orchestra.run.smithi103.stdout:(4/32): libradosstriper1-19.0.0-3189.g8d82defa. 4.0 MB/s | 547 kB 00:00 2024-04-22T06:12:25.591 INFO:teuthology.orchestra.run.smithi028.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-04-22T06:12:25.774 INFO:teuthology.orchestra.run.smithi028.stdout:(27/32): ceph-common-19.0.0-3189.g8d82defa.el9. 6.4 MB/s | 19 MB 00:03 2024-04-22T06:12:25.799 INFO:teuthology.orchestra.run.smithi028.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 919 kB/s | 192 kB 00:00 2024-04-22T06:12:25.825 INFO:teuthology.orchestra.run.smithi103.stdout:(5/32): ceph-radosgw-19.0.0-3189.g8d82defa.el9. 4.9 MB/s | 10 MB 00:02 2024-04-22T06:12:25.892 INFO:teuthology.orchestra.run.smithi103.stdout:(6/32): python3-ceph-argparse-19.0.0-3189.g8d82 679 kB/s | 45 kB 00:00 2024-04-22T06:12:25.959 INFO:teuthology.orchestra.run.smithi103.stdout:(7/32): python3-ceph-common-19.0.0-3189.g8d82de 2.0 MB/s | 135 kB 00:00 2024-04-22T06:12:26.034 INFO:teuthology.orchestra.run.smithi103.stdout:(8/32): librgw2-19.0.0-3189.g8d82defa.el9.x86_6 10 MB/s | 5.3 MB 00:00 2024-04-22T06:12:26.042 INFO:teuthology.orchestra.run.smithi028.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:00 2024-04-22T06:12:26.051 INFO:teuthology.orchestra.run.smithi103.stdout:(9/32): python3-cephfs-19.0.0-3189.g8d82defa.el 1.7 MB/s | 159 kB 00:00 2024-04-22T06:12:26.085 INFO:teuthology.orchestra.run.smithi103.stdout:(10/32): python3-rados-19.0.0-3189.g8d82defa.el 6.2 MB/s | 315 kB 00:00 2024-04-22T06:12:26.127 INFO:teuthology.orchestra.run.smithi103.stdout:(11/32): python3-rbd-19.0.0-3189.g8d82defa.el9. 3.8 MB/s | 293 kB 00:00 2024-04-22T06:12:26.152 INFO:teuthology.orchestra.run.smithi103.stdout:(12/32): python3-rgw-19.0.0-3189.g8d82defa.el9. 1.4 MB/s | 99 kB 00:00 2024-04-22T06:12:26.166 INFO:teuthology.orchestra.run.smithi028.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-04-22T06:12:26.302 INFO:teuthology.orchestra.run.smithi103.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 610 kB/s | 107 kB 00:00 2024-04-22T06:12:26.328 INFO:teuthology.orchestra.run.smithi103.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 914 kB/s | 160 kB 00:00 2024-04-22T06:12:26.350 INFO:teuthology.orchestra.run.smithi028.stdout:(31/32): librados2-19.0.0-3189.g8d82defa.el9.x8 6.2 MB/s | 3.4 MB 00:00 2024-04-22T06:12:26.361 INFO:teuthology.orchestra.run.smithi103.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-22T06:12:26.453 INFO:teuthology.orchestra.run.smithi103.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-22T06:12:26.479 INFO:teuthology.orchestra.run.smithi103.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-22T06:12:26.504 INFO:teuthology.orchestra.run.smithi103.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-04-22T06:12:26.538 INFO:teuthology.orchestra.run.smithi103.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T06:12:26.542 INFO:teuthology.orchestra.run.smithi028.stdout:(32/32): librbd1-19.0.0-3189.g8d82defa.el9.x86_ 7.4 MB/s | 3.7 MB 00:00 2024-04-22T06:12:26.544 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:12:26.545 INFO:teuthology.orchestra.run.smithi028.stdout:Total 15 MB/s | 59 MB 00:04 2024-04-22T06:12:26.554 INFO:teuthology.orchestra.run.smithi103.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 711 kB/s | 36 kB 00:00 2024-04-22T06:12:26.588 INFO:teuthology.orchestra.run.smithi103.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-22T06:12:26.650 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:12:26.691 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:12:26.691 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:12:26.713 INFO:teuthology.orchestra.run.smithi103.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-04-22T06:12:26.747 INFO:teuthology.orchestra.run.smithi103.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 779 kB/s | 26 kB 00:00 2024-04-22T06:12:26.780 INFO:teuthology.orchestra.run.smithi103.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-04-22T06:12:26.856 INFO:teuthology.orchestra.run.smithi103.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-22T06:12:26.881 INFO:teuthology.orchestra.run.smithi103.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-04-22T06:12:26.923 INFO:teuthology.orchestra.run.smithi103.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-22T06:12:26.948 INFO:teuthology.orchestra.run.smithi103.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-22T06:12:26.999 INFO:teuthology.orchestra.run.smithi103.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-22T06:12:27.178 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:12:27.178 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:12:27.250 INFO:teuthology.orchestra.run.smithi103.stdout:(30/32): ceph-common-19.0.0-3189.g8d82defa.el9. 5.5 MB/s | 19 MB 00:03 2024-04-22T06:12:27.350 INFO:teuthology.orchestra.run.smithi103.stdout:(31/32): librados2-19.0.0-3189.g8d82defa.el9.x8 8.5 MB/s | 3.4 MB 00:00 2024-04-22T06:12:27.458 INFO:teuthology.orchestra.run.smithi103.stdout:(32/32): librbd1-19.0.0-3189.g8d82defa.el9.x86_ 8.0 MB/s | 3.7 MB 00:00 2024-04-22T06:12:27.462 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:12:27.462 INFO:teuthology.orchestra.run.smithi103.stdout:Total 15 MB/s | 59 MB 00:03 2024-04-22T06:12:27.562 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:12:27.599 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:12:27.599 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:12:28.061 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:12:28.061 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:12:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:12:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T06:12:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:12:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:12:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:12:29.454 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:12:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:12:29.540 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:12:29.556 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T06:12:29.584 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T06:12:29.609 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T06:12:29.632 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:12:29.656 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:12:29.817 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:12:29.848 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:12:30.111 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T06:12:30.169 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:12:30.218 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T06:12:30.245 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:12:30.277 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:12:30.333 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:12:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:12:30.363 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:12:30.387 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 12/34 2024-04-22T06:12:30.408 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:12:30.444 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:12:30.446 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T06:12:30.475 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T06:12:30.484 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 14/34 2024-04-22T06:12:30.501 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T06:12:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:12:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:12:30.572 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T06:12:30.597 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:12:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:12:30.729 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T06:12:30.750 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:12:30.782 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:12:30.855 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T06:12:30.897 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T06:12:30.937 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T06:12:30.978 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T06:12:31.008 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T06:12:31.038 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T06:12:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T06:12:31.139 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T06:12:31.194 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:12:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:12:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 12/34 2024-04-22T06:12:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:12:31.392 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 14/34 2024-04-22T06:12:31.480 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T06:12:31.505 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:12:31.577 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:12:31.612 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T06:12:31.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T06:12:31.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:12:31.723 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T06:12:31.755 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T06:12:31.775 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:12:31.795 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T06:12:31.820 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T06:12:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T06:12:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 27/34 2024-04-22T06:12:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T06:12:31.889 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 28/34 2024-04-22T06:12:31.917 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T06:12:32.493 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T06:12:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:12:32.591 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:12:32.660 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 27/34 2024-04-22T06:12:32.689 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 28/34 2024-04-22T06:12:34.470 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:12:34.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:12:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:12:35.210 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:12:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:12:35.344 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:12:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:12:35.348 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:12:35.348 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:12:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:12:36.004 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:12:36.064 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:12:36.142 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:12:36.142 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:12:36.142 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:12:36.143 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:36.166 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-22T06:12:45.575 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:46.024 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-22T06:12:46.025 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:12:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:12:46.701 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:12:46.701 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:12:46.701 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:12:46.702 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:12:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:12:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:12:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:12:47.077 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:12:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:12:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:12:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:12:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:12:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:12:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:12:47.187 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:12:47.187 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:12:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:12:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 1/34 2024-04-22T06:12:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:12:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 5/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 6/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 7/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 8/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 9/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 10/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 12/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T06:12:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T06:12:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T06:12:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T06:12:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:12:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T06:12:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 33/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 1/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 5/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 6/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 7/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 8/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 9/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 10/34 2024-04-22T06:12:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 12/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T06:12:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:12:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T06:12:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 33/34 2024-04-22T06:12:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:12:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-22T06:12:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:12:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:12:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:12:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:12:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:12:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:12:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:12:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:48.927 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:12:49.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-test 2024-04-22T06:12:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:12:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:12:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:12:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:12:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:12:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:12:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:12:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:12:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:49.351 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:12:49.664 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-04-22T06:12:49.737 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:29 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:12:49.864 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 84 M 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T06:12:49.865 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 173 k 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout:Install 9 Packages 2024-04-22T06:12:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:12:49.867 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 87 M 2024-04-22T06:12:49.867 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 813 M 2024-04-22T06:12:49.867 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:12:50.234 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:29 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:12:50.358 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 84 M 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 173 k 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:12:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout:Install 9 Packages 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:12:50.360 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 87 M 2024-04-22T06:12:50.361 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 813 M 2024-04-22T06:12:50.361 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:12:50.480 INFO:teuthology.orchestra.run.smithi028.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 489 kB/s | 94 kB 00:00 2024-04-22T06:12:50.691 INFO:teuthology.orchestra.run.smithi028.stdout:(2/9): libcephsqlite-19.0.0-3189.g8d82defa.el9. 430 kB/s | 173 kB 00:00 2024-04-22T06:12:50.766 INFO:teuthology.orchestra.run.smithi028.stdout:(3/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.6 MB/s | 469 kB 00:00 2024-04-22T06:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout:(4/9): libubsan-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-22T06:12:50.926 INFO:teuthology.orchestra.run.smithi028.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-04-22T06:12:50.989 INFO:teuthology.orchestra.run.smithi028.stdout:(6/9): socat-1.7.4.1-5.el9.x86_64.rpm 2.0 MB/s | 305 kB 00:00 2024-04-22T06:12:51.019 INFO:teuthology.orchestra.run.smithi028.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 689 kB/s | 64 kB 00:00 2024-04-22T06:12:51.034 INFO:teuthology.orchestra.run.smithi103.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 588 kB/s | 94 kB 00:00 2024-04-22T06:12:51.154 INFO:teuthology.orchestra.run.smithi028.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 739 kB/s | 122 kB 00:00 2024-04-22T06:12:51.286 INFO:teuthology.orchestra.run.smithi103.stdout:(2/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.8 MB/s | 469 kB 00:00 2024-04-22T06:12:51.370 INFO:teuthology.orchestra.run.smithi103.stdout:(3/9): libubsan-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-04-22T06:12:51.505 INFO:teuthology.orchestra.run.smithi103.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 7.5 MB/s | 1.0 MB 00:00 2024-04-22T06:12:51.530 INFO:teuthology.orchestra.run.smithi103.stdout:(5/9): libcephsqlite-19.0.0-3189.g8d82defa.el9. 264 kB/s | 173 kB 00:00 2024-04-22T06:12:51.589 INFO:teuthology.orchestra.run.smithi103.stdout:(6/9): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-22T06:12:51.732 INFO:teuthology.orchestra.run.smithi103.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 316 kB/s | 64 kB 00:00 2024-04-22T06:12:51.758 INFO:teuthology.orchestra.run.smithi103.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 851 kB/s | 122 kB 00:00 2024-04-22T06:12:55.491 INFO:teuthology.orchestra.run.smithi028.stdout:(9/9): ceph-test-19.0.0-3189.g8d82defa.el9.x86_ 16 MB/s | 84 MB 00:05 2024-04-22T06:12:55.493 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:12:55.493 INFO:teuthology.orchestra.run.smithi028.stdout:Total 15 MB/s | 87 MB 00:05 2024-04-22T06:12:55.576 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:12:55.602 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:12:55.602 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:12:56.229 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:12:56.229 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:12:57.146 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:12:57.206 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T06:12:57.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T06:12:57.305 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T06:12:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T06:12:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:12:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:12:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:12:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T06:12:57.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:12:58.635 INFO:teuthology.orchestra.run.smithi103.stdout:(9/9): ceph-test-19.0.0-3189.g8d82defa.el9.x86_ 11 MB/s | 84 MB 00:07 2024-04-22T06:12:58.639 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:12:58.639 INFO:teuthology.orchestra.run.smithi103.stdout:Total 10 MB/s | 87 MB 00:08 2024-04-22T06:12:58.720 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:12:58.743 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:12:58.744 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:12:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:12:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:13:00.246 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:13:00.317 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T06:13:00.361 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T06:13:00.407 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T06:13:00.432 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T06:13:00.462 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:13:00.496 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:13:00.565 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:13:00.613 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T06:13:00.648 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:13:09.545 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:13:09.620 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:13:11.968 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:13:12.154 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:13:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:13:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 1/9 2024-04-22T06:13:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2/9 2024-04-22T06:13:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T06:13:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T06:13:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:13:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T06:13:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T06:13:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T06:13:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T06:13:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:15.778 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:13:16.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph 2024-04-22T06:13:16.564 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:13:16.693 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:13:16.695 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 6.5 k 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 2.2 M 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 1.6 M 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 249 k 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.0 M 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 71 M 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:13:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:13:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:13:16.698 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout:Install 36 Packages 2024-04-22T06:13:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:16.700 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 84 M 2024-04-22T06:13:16.700 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 923 M 2024-04-22T06:13:16.700 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:13:17.554 INFO:teuthology.orchestra.run.smithi028.stdout:(1/36): ceph-19.0.0-3189.g8d82defa.el9.x86_64.r 31 kB/s | 6.5 kB 00:00 2024-04-22T06:13:17.713 INFO:teuthology.orchestra.run.smithi028.stdout:(2/36): ceph-mgr-19.0.0-3189.g8d82defa.el9.x86_ 4.2 MB/s | 1.6 MB 00:00 2024-04-22T06:13:18.021 INFO:teuthology.orchestra.run.smithi028.stdout:(3/36): ceph-mds-19.0.0-3189.g8d82defa.el9.x86_ 3.2 MB/s | 2.2 MB 00:00 2024-04-22T06:13:18.130 INFO:teuthology.orchestra.run.smithi028.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3189.g8d82 2.2 MB/s | 249 kB 00:00 2024-04-22T06:13:18.517 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:13:18.517 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 1/9 2024-04-22T06:13:18.517 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2/9 2024-04-22T06:13:18.517 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T06:13:18.517 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T06:13:18.517 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:13:18.518 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T06:13:18.518 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T06:13:18.518 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T06:13:18.881 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:13:18.882 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:13:18.882 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:13:18.882 INFO:teuthology.orchestra.run.smithi103.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T06:13:18.882 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:18.882 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:13:19.038 INFO:teuthology.orchestra.run.smithi028.stdout:(5/36): ceph-mon-19.0.0-3189.g8d82defa.el9.x86_ 3.4 MB/s | 5.0 MB 00:01 2024-04-22T06:13:19.063 INFO:teuthology.orchestra.run.smithi028.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 271 kB/s | 253 kB 00:00 2024-04-22T06:13:19.190 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-04-22T06:13:19.704 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:58 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:13:19.797 INFO:teuthology.orchestra.run.smithi028.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-04-22T06:13:19.826 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:13:19.829 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:19.829 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:13:19.829 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:19.829 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:13:19.829 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 6.5 k 2024-04-22T06:13:19.829 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 2.2 M 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 1.6 M 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 249 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.0 M 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 71 M 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:13:19.830 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:13:19.831 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:13:19.832 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:19.833 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:13:19.833 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:19.833 INFO:teuthology.orchestra.run.smithi103.stdout:Install 36 Packages 2024-04-22T06:13:19.833 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:19.833 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 84 M 2024-04-22T06:13:19.834 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 923 M 2024-04-22T06:13:19.834 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:13:20.555 INFO:teuthology.orchestra.run.smithi028.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 178 kB/s | 135 kB 00:00 2024-04-22T06:13:20.666 INFO:teuthology.orchestra.run.smithi103.stdout:(1/36): ceph-19.0.0-3189.g8d82defa.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2024-04-22T06:13:21.224 INFO:teuthology.orchestra.run.smithi103.stdout:(2/36): ceph-mgr-19.0.0-3189.g8d82defa.el9.x86_ 1.9 MB/s | 1.6 MB 00:00 2024-04-22T06:13:21.489 INFO:teuthology.orchestra.run.smithi028.stdout:(9/36): ceph-osd-19.0.0-3189.g8d82defa.el9.x86_ 19 MB/s | 71 MB 00:03 2024-04-22T06:13:21.523 INFO:teuthology.orchestra.run.smithi028.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 505 kB/s | 1.2 MB 00:02 2024-04-22T06:13:21.700 INFO:teuthology.orchestra.run.smithi103.stdout:(3/36): ceph-mon-19.0.0-3189.g8d82defa.el9.x86_ 4.9 MB/s | 5.0 MB 00:01 2024-04-22T06:13:21.750 INFO:teuthology.orchestra.run.smithi103.stdout:(4/36): ceph-mds-19.0.0-3189.g8d82defa.el9.x86_ 1.6 MB/s | 2.2 MB 00:01 2024-04-22T06:13:21.775 INFO:teuthology.orchestra.run.smithi103.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3189.g8d82 3.2 MB/s | 249 kB 00:00 2024-04-22T06:13:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 90 kB/s | 125 kB 00:01 2024-04-22T06:13:22.340 INFO:teuthology.orchestra.run.smithi028.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 253 kB/s | 215 kB 00:00 2024-04-22T06:13:22.365 INFO:teuthology.orchestra.run.smithi028.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 224 kB/s | 188 kB 00:00 2024-04-22T06:13:22.675 INFO:teuthology.orchestra.run.smithi103.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 274 kB/s | 253 kB 00:00 2024-04-22T06:13:22.774 INFO:teuthology.orchestra.run.smithi028.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 207 kB/s | 172 kB 00:00 2024-04-22T06:13:22.940 INFO:teuthology.orchestra.run.smithi028.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2024-04-22T06:13:23.024 INFO:teuthology.orchestra.run.smithi028.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 77 kB/s | 51 kB 00:00 2024-04-22T06:13:23.074 INFO:teuthology.orchestra.run.smithi028.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-04-22T06:13:23.391 INFO:teuthology.orchestra.run.smithi028.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-04-22T06:13:23.409 INFO:teuthology.orchestra.run.smithi103.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-04-22T06:13:23.416 INFO:teuthology.orchestra.run.smithi028.stdout:(19/36): python3-toml-0.10.2-6.el9.noarch.rpm 65 kB/s | 42 kB 00:00 2024-04-22T06:13:23.450 INFO:teuthology.orchestra.run.smithi028.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 380 kB/s | 151 kB 00:00 2024-04-22T06:13:23.559 INFO:teuthology.orchestra.run.smithi028.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-04-22T06:13:23.626 INFO:teuthology.orchestra.run.smithi028.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 733 kB/s | 172 kB 00:00 2024-04-22T06:13:23.651 INFO:teuthology.orchestra.run.smithi028.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-04-22T06:13:23.718 INFO:teuthology.orchestra.run.smithi028.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-04-22T06:13:23.743 INFO:teuthology.orchestra.run.smithi028.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-04-22T06:13:23.810 INFO:teuthology.orchestra.run.smithi028.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-04-22T06:13:23.836 INFO:teuthology.orchestra.run.smithi028.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-04-22T06:13:23.894 INFO:teuthology.orchestra.run.smithi028.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-04-22T06:13:23.917 INFO:teuthology.orchestra.run.smithi103.stdout:(8/36): python3-cryptography-36.0.1-4.el9.x86_6 586 kB/s | 1.2 MB 00:02 2024-04-22T06:13:23.986 INFO:teuthology.orchestra.run.smithi028.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-04-22T06:13:24.012 INFO:teuthology.orchestra.run.smithi028.stdout:(30/36): python3-cherrypy-18.6.1-2.el9.noarch.r 603 kB/s | 358 kB 00:00 2024-04-22T06:13:24.037 INFO:teuthology.orchestra.run.smithi028.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-22T06:13:24.082 INFO:teuthology.orchestra.run.smithi028.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 976 kB/s | 90 kB 00:00 2024-04-22T06:13:24.104 INFO:teuthology.orchestra.run.smithi028.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-04-22T06:13:24.138 INFO:teuthology.orchestra.run.smithi028.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-22T06:13:24.196 INFO:teuthology.orchestra.run.smithi028.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-04-22T06:13:24.201 INFO:teuthology.orchestra.run.smithi103.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 171 kB/s | 135 kB 00:00 2024-04-22T06:13:24.254 INFO:teuthology.orchestra.run.smithi028.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-04-22T06:13:24.257 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:24.257 INFO:teuthology.orchestra.run.smithi028.stdout:Total 11 MB/s | 84 MB 00:07 2024-04-22T06:13:24.402 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:13:24.447 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:13:24.448 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:13:25.095 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:13:25.096 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:13:25.193 INFO:teuthology.orchestra.run.smithi103.stdout:(10/36): ceph-osd-19.0.0-3189.g8d82defa.el9.x86 18 MB/s | 71 MB 00:03 2024-04-22T06:13:25.310 INFO:teuthology.orchestra.run.smithi103.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 90 kB/s | 125 kB 00:01 2024-04-22T06:13:25.444 INFO:teuthology.orchestra.run.smithi103.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 758 kB/s | 188 kB 00:00 2024-04-22T06:13:25.502 INFO:teuthology.orchestra.run.smithi103.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 598 kB/s | 35 kB 00:00 2024-04-22T06:13:25.528 INFO:teuthology.orchestra.run.smithi103.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 797 kB/s | 172 kB 00:00 2024-04-22T06:13:25.570 INFO:teuthology.orchestra.run.smithi103.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 763 kB/s | 51 kB 00:00 2024-04-22T06:13:25.595 INFO:teuthology.orchestra.run.smithi103.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 624 kB/s | 42 kB 00:00 2024-04-22T06:13:25.653 INFO:teuthology.orchestra.run.smithi103.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-04-22T06:13:25.712 INFO:teuthology.orchestra.run.smithi103.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 742 kB/s | 43 kB 00:00 2024-04-22T06:13:25.737 INFO:teuthology.orchestra.run.smithi103.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-04-22T06:13:25.787 INFO:teuthology.orchestra.run.smithi103.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-22T06:13:25.822 INFO:teuthology.orchestra.run.smithi103.stdout:(21/36): python3-urllib3-1.26.5-5.el9.noarch.rp 132 kB/s | 215 kB 00:01 2024-04-22T06:13:25.847 INFO:teuthology.orchestra.run.smithi103.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 185 kB/s | 11 kB 00:00 2024-04-22T06:13:25.864 INFO:teuthology.orchestra.run.smithi103.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-04-22T06:13:25.889 INFO:teuthology.orchestra.run.smithi103.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-04-22T06:13:25.906 INFO:teuthology.orchestra.run.smithi103.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 465 kB/s | 19 kB 00:00 2024-04-22T06:13:25.923 INFO:teuthology.orchestra.run.smithi103.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-04-22T06:13:25.940 INFO:teuthology.orchestra.run.smithi103.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-04-22T06:13:25.957 INFO:teuthology.orchestra.run.smithi103.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-04-22T06:13:25.982 INFO:teuthology.orchestra.run.smithi103.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-04-22T06:13:25.999 INFO:teuthology.orchestra.run.smithi103.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-04-22T06:13:26.024 INFO:teuthology.orchestra.run.smithi103.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-04-22T06:13:26.050 INFO:teuthology.orchestra.run.smithi103.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-04-22T06:13:26.083 INFO:teuthology.orchestra.run.smithi103.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-04-22T06:13:26.108 INFO:teuthology.orchestra.run.smithi103.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-22T06:13:26.134 INFO:teuthology.orchestra.run.smithi103.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-04-22T06:13:26.159 INFO:teuthology.orchestra.run.smithi103.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-04-22T06:13:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:Total 13 MB/s | 84 MB 00:06 2024-04-22T06:13:26.301 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:13:26.341 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:13:26.341 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:13:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:13:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T06:13:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T06:13:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T06:13:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T06:13:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T06:13:26.950 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:13:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T06:13:26.950 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:13:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T06:13:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T06:13:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T06:13:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T06:13:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T06:13:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T06:13:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T06:13:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T06:13:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T06:13:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T06:13:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T06:13:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T06:13:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T06:13:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T06:13:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T06:13:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T06:13:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T06:13:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T06:13:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T06:13:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T06:13:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T06:13:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T06:13:28.512 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:13:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T06:13:28.578 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T06:13:28.610 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T06:13:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T06:13:28.647 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T06:13:28.719 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T06:13:28.771 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T06:13:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T06:13:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 31/36 2024-04-22T06:13:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:13:28.897 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T06:13:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:13:28.906 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:28.907 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:13:28.907 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:13:28.907 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:13:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T06:13:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T06:13:29.059 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T06:13:29.089 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T06:13:29.123 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T06:13:29.165 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T06:13:29.205 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T06:13:29.240 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T06:13:29.274 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T06:13:29.313 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T06:13:29.386 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T06:13:29.534 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T06:13:29.614 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T06:13:29.666 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T06:13:29.705 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T06:13:29.753 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T06:13:29.809 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T06:13:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T06:13:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T06:13:30.128 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T06:13:30.227 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T06:13:30.267 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T06:13:30.382 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T06:13:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 31/36 2024-04-22T06:13:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:13:30.619 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:13:30.620 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:30.620 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:13:30.620 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:13:30.620 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:13:30.620 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:40.389 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:13:40.417 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:13:40.417 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:40.417 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:13:40.417 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:13:40.417 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:13:40.417 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:13:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:13:41.007 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:41.007 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:13:41.007 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:13:41.007 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:13:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:41.257 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:13:41.282 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:13:41.282 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:41.282 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:13:41.283 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:13:41.283 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:13:41.283 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:41.347 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:13:41.411 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:13:41.437 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:13:41.438 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:41.438 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:13:41.438 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:13:41.438 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:13:41.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:41.978 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:13:42.001 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:13:42.001 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:42.002 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:13:42.002 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:13:42.002 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:13:42.002 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:42.246 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:13:42.269 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:13:42.269 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:42.270 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:13:42.270 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:13:42.270 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:13:42.270 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 1/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 3/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 4/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 5/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 6/36 2024-04-22T06:13:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T06:13:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T06:13:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T06:13:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T06:13:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T06:13:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T06:13:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T06:13:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T06:13:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T06:13:48.555 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T06:13:48.555 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:48.555 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:13:48.555 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:48.555 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:48.555 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:13:48.556 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:13:48.557 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:48.558 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:13:48.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-base 2024-04-22T06:13:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:13:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 1/36 2024-04-22T06:13:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2/36 2024-04-22T06:13:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 3/36 2024-04-22T06:13:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 4/36 2024-04-22T06:13:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 5/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 6/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T06:13:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T06:13:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T06:13:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T06:13:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T06:13:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T06:13:49.400 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:13:49.480 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:13:49.524 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:13:49.525 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:13:49.525 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:13:49.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install cephadm 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:13:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:13:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:13:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:50.078 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:13:50.119 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:30 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:13:50.243 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 762 k 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:13:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:50.245 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 762 k 2024-04-22T06:13:50.245 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 769 k 2024-04-22T06:13:50.245 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:13:50.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-04-22T06:13:50.633 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm-19.0.0-3189.g8d82defa.el9.noarch.rpm 1.9 MB/s | 762 kB 00:00 2024-04-22T06:13:50.634 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:50.634 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.9 MB/s | 762 kB 00:00 2024-04-22T06:13:50.634 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:13:50.639 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:13:50.639 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:13:50.645 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:13:50.646 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:13:50.719 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:13:50.933 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:13:51.011 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:13:51.053 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:13:51.054 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:13:51.054 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:13:51.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-04-22T06:13:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:51.287 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:51.626 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:30 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:13:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:13:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:51.746 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 762 k 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 762 k 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 769 k 2024-04-22T06:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:13:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:51.862 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:13:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:13:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:51.862 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:13:52.054 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-19.0.0-3189.g8d82defa.el9.noarch.rpm 2.4 MB/s | 762 kB 00:00 2024-04-22T06:13:52.054 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:52.055 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.4 MB/s | 762 kB 00:00 2024-04-22T06:13:52.055 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:13:52.060 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:13:52.060 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:13:52.065 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:13:52.066 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:13:52.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T06:13:52.134 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:13:52.592 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:52.622 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:13:52.681 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:52.749 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 157 k 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 157 k 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 640 k 2024-04-22T06:13:52.750 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:13:52.973 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-immutable-object-cache-19.0.0-3189.g8d82de 707 kB/s | 157 kB 00:00 2024-04-22T06:13:52.974 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:52.974 INFO:teuthology.orchestra.run.smithi028.stdout:Total 702 kB/s | 157 kB 00:00 2024-04-22T06:13:52.974 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:13:52.985 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:13:52.985 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:13:52.992 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:53.044 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:13:53.045 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:13:53.193 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:13:53.215 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:13:53.216 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:13:53.216 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:13:53.216 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:13:53.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:13:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:13:53.291 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:53.291 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:13:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:53.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T06:13:53.982 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:13:54.103 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:13:54.104 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:54.104 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-22T06:13:54.104 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 157 k 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 157 k 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 640 k 2024-04-22T06:13:54.105 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:13:54.239 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:13:54.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:54.240 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:13:54.240 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:54.240 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:54.240 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:13:54.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr 2024-04-22T06:13:54.603 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-19.0.0-3189.g8d82de 316 kB/s | 157 kB 00:00 2024-04-22T06:13:54.604 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:54.604 INFO:teuthology.orchestra.run.smithi103.stdout:Total 315 kB/s | 157 kB 00:00 2024-04-22T06:13:54.604 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:13:54.614 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:13:54.614 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:13:54.670 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:13:54.671 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:13:54.829 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:13:54.914 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:13:54.937 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:13:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:13:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:13:54.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:54.991 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:13:55.072 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:13:55.117 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:13:55.118 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:13:55.118 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:13:55.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T06:13:55.701 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:13:55.827 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:13:55.827 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:13:55.827 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:55.827 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:13:55.828 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 3.5 M 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 34 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 16 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:55.829 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:13:55.830 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:13:55.830 INFO:teuthology.orchestra.run.smithi028.stdout:Install 11 Packages 2024-04-22T06:13:55.830 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:13:55.830 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 7.6 M 2024-04-22T06:13:55.830 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 100 M 2024-04-22T06:13:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:13:56.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-04-22T06:13:56.565 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:13:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:13:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:13:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:13:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:13:56.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T06:13:56.918 INFO:teuthology.orchestra.run.smithi028.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3189.g8d8 54 kB/s | 16 kB 00:00 2024-04-22T06:13:56.943 INFO:teuthology.orchestra.run.smithi028.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3189.g8d 103 kB/s | 34 kB 00:00 2024-04-22T06:13:57.210 INFO:teuthology.orchestra.run.smithi028.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-04-22T06:13:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:13:57.382 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:13:57.383 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:57.383 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:13:57.383 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:57.383 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:13:57.383 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 3.5 M 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 34 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 16 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:13:57.384 INFO:teuthology.orchestra.run.smithi103.stdout:Install 11 Packages 2024-04-22T06:13:57.385 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:13:57.385 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.6 M 2024-04-22T06:13:57.385 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 100 M 2024-04-22T06:13:57.385 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:13:57.402 INFO:teuthology.orchestra.run.smithi028.stdout:(4/11): ceph-mgr-dashboard-19.0.0-3189.g8d82def 4.5 MB/s | 3.5 MB 00:00 2024-04-22T06:13:57.427 INFO:teuthology.orchestra.run.smithi028.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.8 MB/s | 548 kB 00:00 2024-04-22T06:13:57.486 INFO:teuthology.orchestra.run.smithi028.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-04-22T06:13:57.536 INFO:teuthology.orchestra.run.smithi028.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-04-22T06:13:57.578 INFO:teuthology.orchestra.run.smithi028.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 740 kB/s | 31 kB 00:00 2024-04-22T06:13:57.603 INFO:teuthology.orchestra.run.smithi028.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-04-22T06:13:57.628 INFO:teuthology.orchestra.run.smithi028.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-04-22T06:13:57.954 INFO:teuthology.orchestra.run.smithi028.stdout:(11/11): python3-protobuf-3.14.0-13.el9.noarch. 260 kB/s | 269 kB 00:01 2024-04-22T06:13:57.959 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:57.959 INFO:teuthology.orchestra.run.smithi028.stdout:Total 3.6 MB/s | 7.6 MB 00:02 2024-04-22T06:13:57.993 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:13:58.014 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:13:58.015 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:13:58.175 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:13:58.175 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:13:58.316 INFO:teuthology.orchestra.run.smithi103.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3189.g8d8 84 kB/s | 16 kB 00:00 2024-04-22T06:13:58.424 INFO:teuthology.orchestra.run.smithi103.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3189.g8d 112 kB/s | 34 kB 00:00 2024-04-22T06:13:58.466 INFO:teuthology.orchestra.run.smithi103.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-04-22T06:13:58.567 INFO:teuthology.orchestra.run.smithi103.stdout:(4/11): ceph-mgr-dashboard-19.0.0-3189.g8d82def 7.9 MB/s | 3.5 MB 00:00 2024-04-22T06:13:58.683 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:13:58.771 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T06:13:58.799 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T06:13:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-04-22T06:13:58.834 INFO:teuthology.orchestra.run.smithi103.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-04-22T06:13:58.880 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T06:13:59.016 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T06:13:59.018 INFO:teuthology.orchestra.run.smithi103.stdout:(7/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 995 kB/s | 548 kB 00:00 2024-04-22T06:13:59.078 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T06:13:59.109 INFO:teuthology.orchestra.run.smithi103.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 336 kB/s | 31 kB 00:00 2024-04-22T06:13:59.166 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T06:13:59.201 INFO:teuthology.orchestra.run.smithi103.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-22T06:13:59.214 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T06:13:59.250 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:13:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 9/11 2024-04-22T06:13:59.285 INFO:teuthology.orchestra.run.smithi103.stdout:(10/11): python3-grpcio-tools-1.46.7-10.el9.x86 320 kB/s | 144 kB 00:00 2024-04-22T06:13:59.402 INFO:teuthology.orchestra.run.smithi103.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-04-22T06:13:59.404 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:13:59.404 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.8 MB/s | 7.6 MB 00:02 2024-04-22T06:13:59.435 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:13:59.452 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:13:59.452 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:13:59.597 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:13:59.598 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T06:14:00.189 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T06:14:00.266 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T06:14:00.391 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T06:14:00.396 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 10/11 2024-04-22T06:14:00.434 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T06:14:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:14:00.517 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T06:14:00.570 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T06:14:00.598 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:14:00.631 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 9/11 2024-04-22T06:14:01.676 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 10/11 2024-04-22T06:14:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:14:01.740 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:14:01.740 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 1/11 2024-04-22T06:14:01.740 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 2/11 2024-04-22T06:14:01.740 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 3/11 2024-04-22T06:14:01.740 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T06:14:01.741 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T06:14:01.741 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T06:14:01.741 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T06:14:01.741 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:14:01.741 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T06:14:01.741 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:02.047 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:02.048 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:02.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T06:14:02.816 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:42 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:02.944 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:02.945 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-22T06:14:02.945 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:14:02.945 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-22T06:14:02.945 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:02.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 7.4 M 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:14:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 38 M 2024-04-22T06:14:02.947 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 209 M 2024-04-22T06:14:02.948 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:14:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 1/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 2/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 3/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T06:14:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T06:14:03.253 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2024-04-22T06:14:03.295 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:14:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:14:03.308 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:14:03.308 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:14:03.308 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:03.308 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:03.446 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 171 kB/s | 33 kB 00:00 2024-04-22T06:14:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 313 kB/s | 18 kB 00:00 2024-04-22T06:14:03.562 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-22T06:14:03.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T06:14:03.671 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.0 MB/s | 3.0 MB 00:00 2024-04-22T06:14:03.738 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-22T06:14:03.838 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-04-22T06:14:03.905 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-04-22T06:14:03.989 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-04-22T06:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:43 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:04.204 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:04.205 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-22T06:14:04.205 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 7.4 M 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:14:04.206 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 38 M 2024-04-22T06:14:04.208 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 209 M 2024-04-22T06:14:04.208 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:04.231 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 6.8 MB/s | 7.4 MB 00:01 2024-04-22T06:14:04.439 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-22T06:14:04.442 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:04.443 INFO:teuthology.orchestra.run.smithi028.stdout:Total 25 MB/s | 38 MB 00:01 2024-04-22T06:14:04.610 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 996 kB/s | 195 kB 00:00 2024-04-22T06:14:04.660 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-04-22T06:14:04.778 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-04-22T06:14:04.803 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:04.828 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-04-22T06:14:04.830 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:04.830 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:04.903 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-04-22T06:14:04.928 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 351 kB/s | 35 kB 00:00 2024-04-22T06:14:05.029 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): ceph-mgr-diskprediction-local-19.0.0-31 12 MB/s | 7.4 MB 00:00 2024-04-22T06:14:05.129 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-22T06:14:05.213 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-04-22T06:14:05.266 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:05.266 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:05.355 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-22T06:14:05.422 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 9.1 MB/s | 4.7 MB 00:00 2024-04-22T06:14:05.872 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-22T06:14:05.873 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:05.873 INFO:teuthology.orchestra.run.smithi103.stdout:Total 23 MB/s | 38 MB 00:01 2024-04-22T06:14:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T06:14:06.221 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:14:06.247 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:06.247 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:14:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T06:14:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T06:14:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T06:14:06.653 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:06.654 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T06:14:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T06:14:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T06:14:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T06:14:07.570 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:14:07.680 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:14:07.729 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T06:14:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T06:14:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T06:14:08.013 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T06:14:08.658 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T06:14:08.776 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T06:14:08.783 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:14:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T06:14:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:14:10.023 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:14:11.053 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T06:14:11.080 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:14:12.430 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:14:12.430 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 1/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:14:12.431 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T06:14:12.805 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T06:14:12.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:14:12.806 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:14:12.807 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:14:12.807 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:14:12.807 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:14:12.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:12.807 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:13.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-rook 2024-04-22T06:14:13.685 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:14:13.685 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 1/12 2024-04-22T06:14:13.685 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:14:13.685 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:14:13.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T06:14:13.704 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:53 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:13.830 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 49 k 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:14:13.831 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:13.832 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:13.833 INFO:teuthology.orchestra.run.smithi028.stdout:Install 13 Packages 2024-04-22T06:14:13.833 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:13.833 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.2 M 2024-04-22T06:14:13.833 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 26 M 2024-04-22T06:14:13.833 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:14:14.037 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:14.038 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:14.322 INFO:teuthology.orchestra.run.smithi028.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 72 kB/s | 19 kB 00:00 2024-04-22T06:14:14.347 INFO:teuthology.orchestra.run.smithi028.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 87 kB/s | 26 kB 00:00 2024-04-22T06:14:14.373 INFO:teuthology.orchestra.run.smithi028.stdout:(3/13): ceph-mgr-rook-19.0.0-3189.g8d82defa.el9 155 kB/s | 49 kB 00:00 2024-04-22T06:14:14.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-04-22T06:14:14.739 INFO:teuthology.orchestra.run.smithi028.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 533 kB/s | 222 kB 00:00 2024-04-22T06:14:14.765 INFO:teuthology.orchestra.run.smithi028.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 381 kB/s | 159 kB 00:00 2024-04-22T06:14:14.848 INFO:teuthology.orchestra.run.smithi028.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 494 kB/s | 54 kB 00:00 2024-04-22T06:14:14.957 INFO:teuthology.orchestra.run.smithi028.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 477 kB/s | 279 kB 00:00 2024-04-22T06:14:14.958 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:53 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:14.982 INFO:teuthology.orchestra.run.smithi028.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-04-22T06:14:15.007 INFO:teuthology.orchestra.run.smithi028.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-04-22T06:14:15.088 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:15.089 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:15.089 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:14:15.089 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:15.089 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:15.089 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 49 k 2024-04-22T06:14:15.089 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:15.090 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:15.091 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:15.091 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-04-22T06:14:15.091 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:15.091 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.2 M 2024-04-22T06:14:15.091 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-04-22T06:14:15.091 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:15.149 INFO:teuthology.orchestra.run.smithi028.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-04-22T06:14:15.258 INFO:teuthology.orchestra.run.smithi028.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 673 kB/s | 203 kB 00:00 2024-04-22T06:14:15.283 INFO:teuthology.orchestra.run.smithi028.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-04-22T06:14:15.326 INFO:teuthology.orchestra.run.smithi028.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-04-22T06:14:15.326 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:15.326 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-04-22T06:14:15.397 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:15.413 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:15.414 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:15.507 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:15.508 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:15.508 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 26 kB 00:00 2024-04-22T06:14:15.533 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 19 kB 00:00 2024-04-22T06:14:15.558 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): ceph-mgr-rook-19.0.0-3189.g8d82defa.el9 254 kB/s | 49 kB 00:00 2024-04-22T06:14:15.659 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-22T06:14:15.684 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-04-22T06:14:15.717 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-04-22T06:14:15.776 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-04-22T06:14:15.801 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 275 kB/s | 32 kB 00:00 2024-04-22T06:14:15.818 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 149 kB/s | 15 kB 00:00 2024-04-22T06:14:15.910 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2024-04-22T06:14:15.961 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-04-22T06:14:15.994 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-04-22T06:14:16.027 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-04-22T06:14:16.028 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:16.028 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-04-22T06:14:16.087 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:16.102 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:16.102 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:16.184 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:16.184 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:16.331 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T06:14:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T06:14:16.450 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T06:14:16.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T06:14:16.528 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T06:14:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T06:14:16.692 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T06:14:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T06:14:16.848 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T06:14:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T06:14:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T06:14:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T06:14:17.029 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T06:14:17.123 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T06:14:17.225 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T06:14:17.273 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T06:14:17.572 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T06:14:17.618 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T06:14:17.652 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T06:14:17.702 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T06:14:17.730 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:14:17.994 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T06:14:18.037 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T06:14:18.071 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T06:14:18.118 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T06:14:18.141 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:14:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:14:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 1/13 2024-04-22T06:14:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T06:14:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T06:14:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T06:14:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T06:14:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T06:14:18.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T06:14:18.827 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:18.827 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:18.827 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:18.827 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:14:18.827 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:18.828 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 1/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T06:14:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T06:14:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T06:14:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T06:14:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T06:14:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T06:14:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T06:14:19.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T06:14:19.197 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T06:14:19.197 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:19.197 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:19.197 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:19.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:14:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:19.199 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:19.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T06:14:19.629 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:59 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:19.754 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:19.755 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:14:19.755 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:19.755 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 142 k 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-04-22T06:14:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:19.757 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 7.0 M 2024-04-22T06:14:19.757 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 33 M 2024-04-22T06:14:19.757 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:19.971 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:58 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:20.089 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 142 k 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:14:20.090 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.0 M 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-04-22T06:14:20.091 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:20.153 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.9 MB/s | 248 kB 00:00 2024-04-22T06:14:20.344 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-04-22T06:14:20.528 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-04-22T06:14:20.554 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): ceph-mgr-cephadm-19.0.0-3189.g8d82defa.e 293 kB/s | 142 kB 00:00 2024-04-22T06:14:20.578 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 247 kB/s | 58 kB 00:00 2024-04-22T06:14:20.613 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 900 kB/s | 248 kB 00:00 2024-04-22T06:14:20.629 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2024-04-22T06:14:20.629 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:20.630 INFO:teuthology.orchestra.run.smithi028.stdout:Total 8.0 MB/s | 7.0 MB 00:00 2024-04-22T06:14:20.689 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3189.g8d82defa.e 405 kB/s | 142 kB 00:00 2024-04-22T06:14:20.702 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:20.715 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:20.715 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:20.814 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-22T06:14:20.818 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:20.819 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:21.136 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:21.198 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 133 kB/s | 51 kB 00:00 2024-04-22T06:14:21.223 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 108 kB/s | 58 kB 00:00 2024-04-22T06:14:21.310 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T06:14:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T06:14:21.914 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 421 kB/s | 548 kB 00:01 2024-04-22T06:14:21.918 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:21.918 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.8 MB/s | 7.0 MB 00:01 2024-04-22T06:14:21.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T06:14:21.991 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:22.004 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:22.004 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:22.011 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T06:14:22.082 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T06:14:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:22.109 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:14:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T06:14:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T06:14:22.864 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:14:22.865 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/6 2024-04-22T06:14:22.865 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T06:14:22.865 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T06:14:22.865 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T06:14:22.865 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T06:14:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:23.127 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:23.204 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T06:14:23.265 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T06:14:23.295 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:14:23.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-fuse 2024-04-22T06:14:23.897 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:03 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 871 k 2024-04-22T06:14:24.023 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:24.024 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 951 k 2024-04-22T06:14:24.025 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 4.0 M 2024-04-22T06:14:24.025 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:24.107 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:14:24.107 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/6 2024-04-22T06:14:24.107 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T06:14:24.107 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T06:14:24.107 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T06:14:24.107 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T06:14:24.322 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 639 kB/s | 80 kB 00:00 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:24.353 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:24.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-04-22T06:14:25.097 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): ceph-fuse-19.0.0-3189.g8d82defa.el9.x86_ 967 kB/s | 871 kB 00:00 2024-04-22T06:14:25.098 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:25.098 INFO:teuthology.orchestra.run.smithi028.stdout:Total 886 kB/s | 951 kB 00:01 2024-04-22T06:14:25.111 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:25.120 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:04 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:25.139 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:25.139 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:25.224 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:25.224 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:25.240 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:25.240 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:25.240 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:14:25.240 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 871 k 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-04-22T06:14:25.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:25.242 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 951 k 2024-04-22T06:14:25.242 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 4.0 M 2024-04-22T06:14:25.242 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T06:14:25.625 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 755 kB/s | 80 kB 00:00 2024-04-22T06:14:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:14:25.870 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-19.0.0-3189.g8d82defa.el9.x86_ 2.4 MB/s | 871 kB 00:00 2024-04-22T06:14:25.870 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:25.870 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.5 MB/s | 951 kB 00:00 2024-04-22T06:14:25.880 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:25.904 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:25.904 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:25.986 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:25.987 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:26.184 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:14:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/2 2024-04-22T06:14:26.335 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T06:14:26.415 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:14:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T06:14:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:26.567 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T06:14:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:26.567 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:26.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-volume 2024-04-22T06:14:27.078 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:14:27.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/2 2024-04-22T06:14:27.315 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T06:14:27.316 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:27.316 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:27.316 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T06:14:27.316 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:27.316 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:27.405 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:07 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:27.533 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 273 k 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-04-22T06:14:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:27.535 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 350 k 2024-04-22T06:14:27.535 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 1.5 M 2024-04-22T06:14:27.535 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:27.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2024-04-22T06:14:28.126 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:07 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:28.204 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): ceph-volume-19.0.0-3189.g8d82defa.el9.no 807 kB/s | 273 kB 00:00 2024-04-22T06:14:28.248 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 273 k 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-04-22T06:14:28.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:28.250 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 350 k 2024-04-22T06:14:28.250 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.5 M 2024-04-22T06:14:28.250 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:28.480 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 706 kB/s | 77 kB 00:00 2024-04-22T06:14:28.621 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 102 kB/s | 77 kB 00:00 2024-04-22T06:14:28.621 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:28.622 INFO:teuthology.orchestra.run.smithi028.stdout:Total 322 kB/s | 350 kB 00:01 2024-04-22T06:14:28.635 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:28.646 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:28.647 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:28.676 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:28.677 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:28.764 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-volume-19.0.0-3189.g8d82defa.el9.no 695 kB/s | 273 kB 00:00 2024-04-22T06:14:28.764 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:28.764 INFO:teuthology.orchestra.run.smithi103.stdout:Total 680 kB/s | 350 kB 00:00 2024-04-22T06:14:28.775 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:28.811 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:28.811 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T06:14:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:14:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:14:29.064 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:14:29.064 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:14:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:29.144 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T06:14:29.183 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:14:29.198 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:14:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:14:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:14:29.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:29.737 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 1/2 2024-04-22T06:14:29.879 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 1/2 2024-04-22T06:14:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T06:14:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:14:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:30.158 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T06:14:30.159 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:30.159 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:30.159 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:14:30.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:14:30.159 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:30.159 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:30.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados-devel 2024-04-22T06:14:30.469 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-04-22T06:14:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:10 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 125 k 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:30.979 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:14:30.980 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:30.980 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 125 k 2024-04-22T06:14:30.980 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 445 k 2024-04-22T06:14:30.980 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:30.991 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:09 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:31.117 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 125 k 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:14:31.119 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:31.119 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 125 k 2024-04-22T06:14:31.119 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 445 k 2024-04-22T06:14:31.119 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:31.183 INFO:teuthology.orchestra.run.smithi028.stdout:librados-devel-19.0.0-3189.g8d82defa.el9.x86_64 611 kB/s | 125 kB 00:00 2024-04-22T06:14:31.184 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:31.184 INFO:teuthology.orchestra.run.smithi028.stdout:Total 606 kB/s | 125 kB 00:00 2024-04-22T06:14:31.184 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:31.190 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:31.190 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:31.257 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:31.257 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:31.410 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:31.500 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:31.915 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:32.128 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-19.0.0-3189.g8d82defa.el9.x86_64 124 kB/s | 125 kB 00:01 2024-04-22T06:14:32.128 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:32.128 INFO:teuthology.orchestra.run.smithi103.stdout:Total 124 kB/s | 125 kB 00:01 2024-04-22T06:14:32.129 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:32.135 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:32.135 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:32.175 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:32.175 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:32.175 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:32.176 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:32.176 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:32.176 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:32.202 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:32.202 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:32.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs2 2024-04-22T06:14:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:32.956 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:12 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:Package libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:33.083 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:33.084 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:33.084 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:33.102 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:33.102 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:33.102 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:33.102 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.102 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:33.102 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:33.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs-devel 2024-04-22T06:14:33.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-04-22T06:14:33.679 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:13 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:33.804 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 32 k 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:33.805 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:33.806 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:14:33.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:33.806 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 32 k 2024-04-22T06:14:33.806 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 142 k 2024-04-22T06:14:33.806 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:33.834 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:12 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:33.912 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:33.954 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:33.955 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:33.955 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:33.978 INFO:teuthology.orchestra.run.smithi028.stdout:libcephfs-devel-19.0.0-3189.g8d82defa.el9.x86_6 184 kB/s | 32 kB 00:00 2024-04-22T06:14:33.978 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:33.978 INFO:teuthology.orchestra.run.smithi028.stdout:Total 183 kB/s | 32 kB 00:00 2024-04-22T06:14:33.979 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:33.982 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:33.982 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:33.984 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:33.985 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:34.023 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-04-22T06:14:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:34.538 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:13 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:34.559 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:34.659 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 32 k 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:34.660 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 32 k 2024-04-22T06:14:34.661 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 142 k 2024-04-22T06:14:34.661 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:34.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:34.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:34.773 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:34.773 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:34.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:34.773 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:34.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados2 2024-04-22T06:14:35.073 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-19.0.0-3189.g8d82defa.el9.x86_6 77 kB/s | 32 kB 00:00 2024-04-22T06:14:35.074 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:35.074 INFO:teuthology.orchestra.run.smithi103.stdout:Total 76 kB/s | 32 kB 00:00 2024-04-22T06:14:35.074 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:35.077 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:35.078 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:35.080 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:35.080 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:35.167 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:35.269 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:35.498 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:15 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:35.580 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:35.607 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:35.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librbd1 2024-04-22T06:14:35.817 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:35.817 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:35.817 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:35.817 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:35.817 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:35.818 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:36.007 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-04-22T06:14:36.224 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:36.306 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:36.352 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:36.352 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:36.353 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:36.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rados 2024-04-22T06:14:36.528 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:15 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:36.606 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:36.646 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:36.647 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:36.648 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:36.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-04-22T06:14:36.948 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:37.031 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:37.075 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:37.076 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:37.076 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:37.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rgw 2024-04-22T06:14:37.225 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:37.304 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:37.346 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:37.347 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:37.347 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:37.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-04-22T06:14:37.672 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:17 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:37.755 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:37.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-cephfs 2024-04-22T06:14:37.922 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:38.000 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:38.041 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:38.042 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:38.042 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:38.109 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-04-22T06:14:38.408 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:18 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:38.491 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:38.534 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:38.537 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:38.537 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:38.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rbd 2024-04-22T06:14:38.619 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:17 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:38.698 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:38.739 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:38.740 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:38.740 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:38.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-04-22T06:14:39.135 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:39.215 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:39.259 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:39.260 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-22T06:14:39.260 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:39.321 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:18 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:39.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-fuse 2024-04-22T06:14:39.400 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:39.442 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:39.443 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:39.506 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-04-22T06:14:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:39.980 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:39.981 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:39.981 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-22T06:14:39.981 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:39.981 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:39.981 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 90 k 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 90 k 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 313 k 2024-04-22T06:14:39.982 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:40.091 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:14:40.131 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:40.132 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-22T06:14:40.133 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:40.197 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-fuse-19.0.0-3189.g8d82defa.el9.x86_64.rpm 418 kB/s | 90 kB 00:00 2024-04-22T06:14:40.198 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:40.198 INFO:teuthology.orchestra.run.smithi028.stdout:Total 416 kB/s | 90 kB 00:00 2024-04-22T06:14:40.198 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:40.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-04-22T06:14:40.208 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:40.208 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:40.275 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:40.275 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:40.422 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:40.711 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:40.830 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:40.831 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-22T06:14:40.831 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:40.831 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 90 k 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 90 k 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 313 k 2024-04-22T06:14:40.832 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:40.893 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:41.139 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:41.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:41.139 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:41.139 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:41.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:41.139 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:41.159 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-19.0.0-3189.g8d82defa.el9.x86_64.rpm 275 kB/s | 90 kB 00:00 2024-04-22T06:14:41.160 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:41.160 INFO:teuthology.orchestra.run.smithi103.stdout:Total 274 kB/s | 90 kB 00:00 2024-04-22T06:14:41.160 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:41.169 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:41.169 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:41.232 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:41.232 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:41.372 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:41.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-mirror 2024-04-22T06:14:41.489 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:21 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:42.091 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:42.092 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:14:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.4 M 2024-04-22T06:14:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 29 M 2024-04-22T06:14:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:42.296 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:42.296 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:42.296 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:42.296 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:42.297 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:42.297 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:42.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-04-22T06:14:42.492 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-mirror-19.0.0-3189.g8d82defa.el9.x86_64.rpm 8.5 MB/s | 3.4 MB 00:00 2024-04-22T06:14:42.493 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:42.493 INFO:teuthology.orchestra.run.smithi028.stdout:Total 8.5 MB/s | 3.4 MB 00:00 2024-04-22T06:14:42.493 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:42.506 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:42.506 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:42.598 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:42.598 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:42.870 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:21 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:42.993 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:42.993 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:42.993 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:14:42.994 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:42.995 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.4 M 2024-04-22T06:14:42.995 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 29 M 2024-04-22T06:14:42.995 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:14:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:14:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:14:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:14:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:43.405 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-19.0.0-3189.g8d82defa.el9.x86_64.rpm 8.3 MB/s | 3.4 MB 00:00 2024-04-22T06:14:43.419 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:43.419 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.3 MB/s | 3.4 MB 00:00 2024-04-22T06:14:43.420 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:43.422 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:43.422 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:43.504 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:44.072 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:44.130 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:44.156 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:44.156 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:14:44.156 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:14:44.156 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:14:44.156 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:14:44.156 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:44.413 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:44.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:44.413 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:44.413 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:44.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:44.414 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:44.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-nbd 2024-04-22T06:14:45.187 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 06:12:20 AM UTC. 2024-04-22T06:14:45.312 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 175 k 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-22T06:14:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-04-22T06:14:45.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:45.314 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 175 k 2024-04-22T06:14:45.314 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 673 k 2024-04-22T06:14:45.314 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-22T06:14:45.331 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:45.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:45.331 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:45.331 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:45.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:45.331 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:45.544 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-nbd-19.0.0-3189.g8d82defa.el9.x86_64.rpm 761 kB/s | 175 kB 00:00 2024-04-22T06:14:45.545 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:45.545 INFO:teuthology.orchestra.run.smithi028.stdout:Total 757 kB/s | 175 kB 00:00 2024-04-22T06:14:45.545 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-22T06:14:45.556 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-22T06:14:45.556 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-22T06:14:45.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-04-22T06:14:45.625 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-22T06:14:45.625 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-22T06:14:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-22T06:14:45.885 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:46.109 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 06:12:21 AM UTC. 2024-04-22T06:14:46.233 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-22T06:14:46.233 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:46.233 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 175 k 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 175 k 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 673 k 2024-04-22T06:14:46.234 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-22T06:14:46.307 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:46.538 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:46.538 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:46.538 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-22T06:14:46.538 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:46.538 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:46.538 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-22T06:14:46.542 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-19.0.0-3189.g8d82defa.el9.x86_64.rpm 571 kB/s | 175 kB 00:00 2024-04-22T06:14:46.543 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:46.543 INFO:teuthology.orchestra.run.smithi103.stdout:Total 568 kB/s | 175 kB 00:00 2024-04-22T06:14:46.543 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-22T06:14:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-22T06:14:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-22T06:14:46.618 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-22T06:14:46.619 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-22T06:14:46.763 DEBUG:teuthology.parallel:result is None 2024-04-22T06:14:46.775 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-22T06:14:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:47.283 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:47.509 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:14:47.509 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-22T06:14:47.509 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:47.509 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-22T06:14:47.752 DEBUG:teuthology.parallel:result is None 2024-04-22T06:14:47.753 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:14:47.934 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:14:47.960 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-3189.g8d82defa.el9 2024-04-22T06:14:47.961 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3189.g8d82defa.el9 2024-04-22T06:14:47.961 INFO:teuthology.task.install:The correct ceph version 19.0.0-3189.g8d82defa is installed. 2024-04-22T06:14:47.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:14:48.110 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:14:48.132 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-3189.g8d82defa.el9 2024-04-22T06:14:48.132 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3189.g8d82defa.el9 2024-04-22T06:14:48.132 INFO:teuthology.task.install:The correct ceph version 19.0.0-3189.g8d82defa is installed. 2024-04-22T06:14:48.135 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T06:14:48.135 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:48.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:14:48.166 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:48.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:14:48.197 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T06:14:48.197 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:48.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:14:48.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:14:48.303 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:48.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:14:48.327 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:14:48.391 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T06:14:48.391 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:48.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:14:48.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:14:48.486 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:48.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:14:48.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:14:48.579 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T06:14:48.579 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:48.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:14:48.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:14:48.672 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:48.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:14:48.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:14:48.759 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T06:14:48.856 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T06:14:48.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-04-22T06:14:48.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-04-22T06:14:48.885 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T06:14:48.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T06:14:48.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T06:14:48.953 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T06:14:48.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T06:14:48.996 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T06:14:49.030 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T06:14:49.030 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T06:14:49.030 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179', 'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7668179', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCArwe8JY69vRPI1gOppoAfuoqH5tvZqOUAgJncBEpBp/SABpASB1UDcRygqcH3J18xC9OFcHZCwRO4oq2Y24ak=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGX9O9cuJkHoW43qGAdaoeyYRTcsLwDfmJ7iTXZTE/RP482xLx50nETZMnXxLAlwMqOuk7JBZyCUl9Qk+TYaugA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'radosbench': {'clients': ['client.0'], 'time': 150}}, {'admin_socket': {'client.0': {'objecter_requests': {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-22_05:57:37', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T06:14:49.031 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T06:14:49.063 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T06:14:49.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T06:14:49.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T06:14:49.145 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:49.145 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:14:49.202 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:14:49.202 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-22T06:14:49.258 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:14:49.258 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:49.258 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T06:14:49.259 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:49.259 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:49.259 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-22 06:14:43.526654458 +0000 2024-04-22T06:14:49.259 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-22 06:11:26.530971096 +0000 2024-04-22T06:14:49.259 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-22 06:11:26.530971096 +0000 2024-04-22T06:14:49.259 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-22 06:11:26.530971096 +0000 2024-04-22T06:14:49.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:14:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-22T06:14:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-22T06:14:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000107047 s, 4.8 MB/s 2024-04-22T06:14:49.324 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:14:49.382 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-22 06:14:43.526654458 +0000 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-22 06:11:26.875961413 +0000 2024-04-22T06:14:49.439 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-22 06:11:26.875961413 +0000 2024-04-22T06:14:49.440 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-22 06:11:26.875961413 +0000 2024-04-22T06:14:49.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:14:49.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-22T06:14:49.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-22T06:14:49.504 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000102818 s, 5.0 MB/s 2024-04-22T06:14:49.505 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:14:49.561 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-22T06:14:49.617 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:14:49.617 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:49.617 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-22T06:14:49.618 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:49.618 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:49.618 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-22 06:14:43.526654458 +0000 2024-04-22T06:14:49.618 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-22 06:11:27.210952011 +0000 2024-04-22T06:14:49.618 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-22 06:11:27.210952011 +0000 2024-04-22T06:14:49.618 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-22 06:11:27.210952011 +0000 2024-04-22T06:14:49.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:14:49.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-22T06:14:49.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-22T06:14:49.681 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.8721e-05 s, 5.8 MB/s 2024-04-22T06:14:49.682 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:14:49.738 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-22 06:14:43.527654430 +0000 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-22 06:11:27.545942610 +0000 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-22 06:11:27.545942610 +0000 2024-04-22T06:14:49.795 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-22 06:11:27.545942610 +0000 2024-04-22T06:14:49.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:14:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-22T06:14:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-22T06:14:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000105418 s, 4.9 MB/s 2024-04-22T06:14:49.859 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:14:49.916 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-22T06:14:49.916 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:49.917 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:14:49.933 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:14:49.934 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-22T06:14:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:14:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:49.989 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T06:14:49.989 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:49.989 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:49.989 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-22 06:14:44.402316917 +0000 2024-04-22T06:14:49.990 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-22 06:11:23.035260731 +0000 2024-04-22T06:14:49.990 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-22 06:11:23.035260731 +0000 2024-04-22T06:14:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-22 06:11:23.035260731 +0000 2024-04-22T06:14:49.990 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:14:50.053 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-22T06:14:50.053 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-22T06:14:50.053 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000214662 s, 2.4 MB/s 2024-04-22T06:14:50.055 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:14:50.111 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-22 06:14:44.402316917 +0000 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-22 06:11:23.385254141 +0000 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-22 06:11:23.385254141 +0000 2024-04-22T06:14:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-22 06:11:23.385254141 +0000 2024-04-22T06:14:50.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:14:50.230 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-22T06:14:50.230 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-22T06:14:50.230 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000211126 s, 2.4 MB/s 2024-04-22T06:14:50.231 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:14:50.287 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-22 06:14:44.403316898 +0000 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-22 06:11:23.738247495 +0000 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-22 06:11:23.738247495 +0000 2024-04-22T06:14:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-22 06:11:23.738247495 +0000 2024-04-22T06:14:50.344 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:14:50.406 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-22T06:14:50.406 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-22T06:14:50.406 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00018727 s, 2.7 MB/s 2024-04-22T06:14:50.407 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:14:50.463 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-22T06:14:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:14:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:14:50.519 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T06:14:50.519 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:14:50.519 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:14:50.520 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-22 06:14:44.403316898 +0000 2024-04-22T06:14:50.520 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-22 06:11:24.083240999 +0000 2024-04-22T06:14:50.520 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-22 06:11:24.083240999 +0000 2024-04-22T06:14:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-22 06:11:24.083240999 +0000 2024-04-22T06:14:50.520 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:14:50.580 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-22T06:14:50.581 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-22T06:14:50.581 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000174944 s, 2.9 MB/s 2024-04-22T06:14:50.582 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:14:50.637 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-22T06:14:50.638 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-22T06:14:50.638 INFO:tasks.ceph:Generating config... 2024-04-22T06:14:50.639 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-22T06:14:50.639 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T06:14:50.639 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] log to file = False 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T06:14:50.640 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-22T06:14:50.641 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-22T06:14:50.641 INFO:tasks.ceph:[client.0] admin socket = /var/run/ceph/ceph-$name.asok 2024-04-22T06:14:50.641 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:14:50.641 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-22T06:14:50.642 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-22T06:14:50.642 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-22T06:14:50.643 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-22T06:14:50.643 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-22T06:14:50.643 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-22T06:14:50.643 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-22T06:14:50.643 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-22T06:14:50.643 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-22T06:14:50.643 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-22T06:14:50.643 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T06:14:50.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T06:14:50.711 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T06:14:50.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T06:14:50.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:14:50.796 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28')] 2024-04-22T06:14:50.796 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-22T06:14:50.797 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:50.797 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T06:14:50.853 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:14:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:14:50.946+0000 7f7edd267f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:14:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:14:50.946+0000 7f7edd267f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:14:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:14:50.946+0000 7f7edd267f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:14:50.948 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-04-22T06:14:50.947678+0000 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-22T06:14:50.947678+0000 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 17 (quincy) 2024-04-22T06:14:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-04-22T06:14:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-22T06:14:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-22T06:14:50.950 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T06:14:50.966 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 507cb479-0568-4196-be48-f7b4ebc16539... 2024-04-22T06:14:50.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T06:14:51.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T06:14:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T06:14:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = 172.21.15.28 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: ms client mode = crc 2024-04-22T06:14:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: ms cluster mode = crc 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: ms mon client mode = crc 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: ms mon cluster mode = crc 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: ms mon service mode = crc 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: ms service mode = crc 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_min_size = 2 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_size = 3 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 120 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-04-22T06:14:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-04-22T06:14:51.055 INFO:teuthology.orchestra.run.smithi028.stdout: debug monc = 20 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore compression algorithm = snappy 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore compression mode = aggressive 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore rocksdb cf = False 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: crimson osd obc lru size = 10 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: err to stderr = True 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: log flush on exit = True 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: log to file = False 2024-04-22T06:14:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: log to stderr = True 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd blocked scrub grace period = 3600 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug reject backfill probability = 0.3 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd delete sleep = 1 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd max backfills = 3 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd max markdown count = 1000 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub min interval = 60 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd shutdown pgref assert = True 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: osd snap trim sleep = 2 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.057 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-04-22T06:14:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon min osdmap epochs = 50 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon osdmap full prune interval = 2 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon osdmap full prune min = 15 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: mon osdmap full prune txsize = 2 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default crimson = True 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: paxos service trim min = 10 2024-04-22T06:14:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:14:51.061 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-04-22T06:14:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:14:51.062 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-04-22T06:14:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:14:51.062 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-04-22T06:14:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-22T06:14:51.064 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-22T06:14:51.065 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-22T06:14:51.066 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = 172.21.15.28 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: ms client mode = crc 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: ms cluster mode = crc 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon client mode = crc 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon cluster mode = crc 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon service mode = crc 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: ms service mode = crc 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_min_size = 2 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_size = 3 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-22T06:14:51.067 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 120 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-22T06:14:51.068 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-22T06:14:51.069 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 20 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore compression algorithm = snappy 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore compression mode = aggressive 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore rocksdb cf = False 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: crimson osd obc lru size = 10 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-04-22T06:14:51.070 INFO:teuthology.orchestra.run.smithi103.stdout: err to stderr = True 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: log flush on exit = True 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: log to file = False 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: log to stderr = True 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: osd blocked scrub grace period = 3600 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug reject backfill probability = 0.3 2024-04-22T06:14:51.071 INFO:teuthology.orchestra.run.smithi103.stdout: osd delete sleep = 1 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd max backfills = 3 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd max markdown count = 1000 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub min interval = 60 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd shutdown pgref assert = True 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: osd snap trim sleep = 2 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-22T06:14:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-22T06:14:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T06:14:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: mon min osdmap epochs = 50 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune interval = 2 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune min = 15 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune txsize = 2 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default crimson = True 2024-04-22T06:14:51.075 INFO:teuthology.orchestra.run.smithi103.stdout: paxos service trim min = 10 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-04-22T06:14:51.076 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-04-22T06:14:51.077 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:14:51.077 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T06:14:51.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T06:14:51.169 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T06:14:51.169 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:51.169 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T06:14:51.185 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:51.185 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T06:14:51.240 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T06:14:51.240 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:51.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T06:14:51.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:14:51.314 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:14:51.314 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:14:51.369 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T06:14:51.369 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:51.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T06:14:51.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:14:51.402 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:14:51.402 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:14:51.457 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T06:14:51.457 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T06:14:51.457 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T06:14:51.559 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T06:14:51.562 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T06:14:51.562 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T06:14:51.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T06:14:51.610 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T06:14:51.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T06:14:51.673 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-22T06:14:51.686 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T06:14:51.686 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-22T06:14:51.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T06:14:51.710 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-22T06:14:51.711 INFO:tasks.ceph:role: osd.0 2024-04-22T06:14:51.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T06:14:51.711 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T06:14:51.775 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:14:51.775 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:14:51.776 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:14:51.807 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-04-22T06:14:51.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-04-22T06:14:51.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T06:14:51.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T06:14:51.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T06:14:52.018 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-22T06:14:52.018 INFO:tasks.ceph:role: osd.1 2024-04-22T06:14:52.018 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T06:14:52.018 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:14:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:14:52.084 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:14:52.116 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-04-22T06:14:52.118 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-04-22T06:14:52.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T06:14:52.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T06:14:52.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:14:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:14:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:52,592 seastar - Reactor backend: linux-aio 2024-04-22T06:14:52.609 INFO:teuthology.orchestra.run.smithi028.stderr:==26390==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:14:52.714 INFO:teuthology.orchestra.run.smithi028.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x5e3a2c6 0x5e3a469 0x5db8f50 0x5db9484 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x37e091a 0x37e1473 0x3680d7a 0x2958f 0x2963f 0x346d444 2024-04-22T06:14:52.714 INFO:teuthology.orchestra.run.smithi028.stderr:kernel callstack: 2024-04-22T06:14:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:52,735 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T06:14:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:52,735 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-22T06:14:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:DEBUG 2024-04-22 06:14:52,736 [shard 0:main] seastar - reactor::drain 2024-04-22T06:14:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:14:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:Suppressions used: 2024-04-22T06:14:53.297 INFO:teuthology.orchestra.run.smithi028.stderr: count bytes template 2024-04-22T06:14:53.297 INFO:teuthology.orchestra.run.smithi028.stderr: 1 8 MallocExtension::Register 2024-04-22T06:14:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:14:53.297 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T06:14:53.359 INFO:teuthology.orchestra.run.smithi028.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:14:53.363 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:53,364 seastar - Reactor backend: linux-aio 2024-04-22T06:14:53.372 INFO:teuthology.orchestra.run.smithi028.stderr:==26386==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:14:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x4a06c 0x35b7c 0x3629e 0x5deb354 0x5deb536 0x5deb74d 0x5deba23 0x5e3a263 0x5e3a469 0x5db8f50 0x5db912a 0x5db96b6 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x368158a 0x2958f 0x2963f 0x346d444 2024-04-22T06:14:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:kernel callstack: 2024-04-22T06:14:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:14:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:Suppressions used: 2024-04-22T06:14:58.037 INFO:teuthology.orchestra.run.smithi028.stderr: count bytes template 2024-04-22T06:14:58.037 INFO:teuthology.orchestra.run.smithi028.stderr: 1 8 MallocExtension::Register 2024-04-22T06:14:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:14:58.037 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T06:14:58.038 INFO:teuthology.orchestra.run.smithi028.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:14:58.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T06:14:58.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:14:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:14:58.391 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:58,392 seastar - Reactor backend: linux-aio 2024-04-22T06:14:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:==26573==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:14:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x34c760f 0x36b50ee 0x3fed0e5 0x5db8f2e 0x5db912a 0x5db96b6 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x37e091a 0x37e1473 0x3680d7a 0x2958f 0x2963f 0x346d444 2024-04-22T06:14:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:kernel callstack: 2024-04-22T06:14:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:58,520 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T06:14:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:58,520 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-22T06:14:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:DEBUG 2024-04-22 06:14:58,521 [shard 0:main] seastar - reactor::drain 2024-04-22T06:14:59.080 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:14:59.080 INFO:teuthology.orchestra.run.smithi028.stderr:Suppressions used: 2024-04-22T06:14:59.080 INFO:teuthology.orchestra.run.smithi028.stderr: count bytes template 2024-04-22T06:14:59.080 INFO:teuthology.orchestra.run.smithi028.stderr: 1 8 MallocExtension::Register 2024-04-22T06:14:59.080 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:14:59.080 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T06:14:59.138 INFO:teuthology.orchestra.run.smithi028.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:14:59.142 INFO:teuthology.orchestra.run.smithi028.stderr:INFO 2024-04-22 06:14:59,143 seastar - Reactor backend: linux-aio 2024-04-22T06:14:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:==26571==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:14:59.216 INFO:teuthology.orchestra.run.smithi028.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x4a0ae 0x35b7c 0x3629e 0x5deb354 0x5deb51c 0x5deb74d 0x5deba23 0x5e3a263 0x5e3a469 0x5db8f50 0x5db9484 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x368158a 0x2958f 0x2963f 0x346d444 2024-04-22T06:14:59.216 INFO:teuthology.orchestra.run.smithi028.stderr:kernel callstack: 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stderr:Suppressions used: 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stderr: count bytes template 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stderr: 1 8 MallocExtension::Register 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stderr:----------------------------------------------------- 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T06:15:03.826 INFO:teuthology.orchestra.run.smithi028.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:15:03.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T06:15:03.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T06:15:03.902 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-22T06:15:03.903 INFO:tasks.ceph:role: osd.2 2024-04-22T06:15:03.903 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T06:15:03.903 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:15:03.977 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:15:03.978 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:15:03.978 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:15:03.978 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:15:04.484 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-22T06:15:04.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-22T06:15:04.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-22T06:15:04.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T06:15:04.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-22T06:15:04.729 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-22T06:15:04.729 INFO:tasks.ceph:role: osd.3 2024-04-22T06:15:04.729 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T06:15:04.729 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T06:15:04.801 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:15:04.802 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:15:05.266 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-22T06:15:05.267 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-22T06:15:05.268 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-22T06:15:05.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-22T06:15:05.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:15:05.712 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:15:05.713 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:05,714 seastar - Reactor backend: linux-aio 2024-04-22T06:15:05.725 INFO:teuthology.orchestra.run.smithi103.stderr:==26477==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:15:05.841 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:05,842 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T06:15:05.842 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:05,842 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-22T06:15:05.842 INFO:teuthology.orchestra.run.smithi103.stderr:DEBUG 2024-04-22 06:15:05,843 [shard 0:main] seastar - reactor::drain 2024-04-22T06:15:06.385 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:06.385 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-22T06:15:06.385 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-22T06:15:06.385 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-22T06:15:06.385 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:06.385 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-22T06:15:06.442 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:15:06.445 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:06,446 seastar - Reactor backend: linux-aio 2024-04-22T06:15:06.454 INFO:teuthology.orchestra.run.smithi103.stderr:==26475==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:15:11.075 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:11.076 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-22T06:15:11.076 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-22T06:15:11.076 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-22T06:15:11.076 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:11.076 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-22T06:15:11.076 INFO:teuthology.orchestra.run.smithi103.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:15:11.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T06:15:11.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:15:11.401 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:15:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:11,403 seastar - Reactor backend: linux-aio 2024-04-22T06:15:11.407 INFO:teuthology.orchestra.run.smithi103.stderr:==26660==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:15:11.518 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:11,519 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T06:15:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:11,519 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-22T06:15:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:DEBUG 2024-04-22 06:15:11,520 [shard 0:main] seastar - reactor::drain 2024-04-22T06:15:12.059 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:12.060 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-22T06:15:12.060 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-22T06:15:12.060 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-22T06:15:12.060 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:12.060 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-22T06:15:12.115 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:15:12.119 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-22 06:15:12,120 seastar - Reactor backend: linux-aio 2024-04-22T06:15:12.134 INFO:teuthology.orchestra.run.smithi103.stderr:==26658==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-22T06:15:16.795 INFO:teuthology.orchestra.run.smithi103.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:15:16.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T06:15:16.840 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T06:15:16.840 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:15:16.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T06:15:16.870 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:15:16.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T06:15:16.939 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:15:16.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T06:15:16.963 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:15:16.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T06:15:17.026 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:15:17.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-22T06:15:17.088 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:15:17.089 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T06:15:17.107 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:15:17.107 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-22T06:15:17.143 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T06:15:17.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBdACZmM22GHxAAtcKB4S1ijtVksGlgUwDOgg== 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBjACZm3G/fERAArIh+vpqcAz4QDj419v8qgQ== 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBbACZmcI5lIRAAfpvAR71bUNFTkA2dnGAReQ== 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-04-22T06:15:17.175 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBqACZmkt1rIxAA7brkKyVfYn7K+JnEjZn1RA== 2024-04-22T06:15:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-04-22T06:15:17.176 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBwACZmOCgJEBAAcxKk80oXgANB3/Fe+jsTng== 2024-04-22T06:15:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-22T06:15:17.176 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBbACZmCrJ7JBAADm1/NUpMjf36JD7HhrCEkg== 2024-04-22T06:15:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-04-22T06:15:17.176 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBbACZmCzc3KBAALXH604afJ55XjExw00yfoA== 2024-04-22T06:15:17.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:15:17.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:15:17.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T06:15:17.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:15:17.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:15:17.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T06:15:17.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T06:15:17.657 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T06:15:17.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T06:15:17.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T06:15:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:17.794+0000 7f2a3a836a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:17.794+0000 7f2a3a836a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:17.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T06:15:17.832 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:15:17.887 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T06:15:17.888 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T06:15:17.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T06:15:17.929 INFO:tasks.ceph.mon.a:Started 2024-04-22T06:15:17.929 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T06:15:17.929 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T06:15:17.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T06:15:17.932 INFO:tasks.ceph.mgr.x:Started 2024-04-22T06:15:17.932 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T06:15:17.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T06:15:18.012 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-22T06:15:18.011+0000 7f4515c30a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.012 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-22T06:15:18.011+0000 7f4515c30a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.012 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-22T06:15:18.012+0000 7f4515c30a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.040 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.039+0000 7fd5df234000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:18.037+0000 7f8d3b006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.045 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.044+0000 7fd5df234000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.045 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.044+0000 7fd5df234000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:18.048+0000 7f8d3b006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-22T06:15:18.134 INFO:tasks.ceph:check_enable_crimson: True 2024-04-22T06:15:18.134 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-22T06:15:18.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-22T06:15:18.152 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.152+0000 7fd5df234000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:15:18.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:18.238+0000 7fad96ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:18.247+0000 7fad96ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:18.306 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.305+0000 7fd5df234000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:15:18.331 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-22T06:15:18.331+0000 7f450c5f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-22T06:15:18.331 INFO:tasks.ceph.mon.a.smithi028.stderr:Please be aware that this feature is experimental, untested, 2024-04-22T06:15:18.331 INFO:tasks.ceph.mon.a.smithi028.stderr:unsupported, and may result in data corruption, data loss, 2024-04-22T06:15:18.331 INFO:tasks.ceph.mon.a.smithi028.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-22T06:15:18.331 INFO:tasks.ceph.mon.a.smithi028.stderr:feature with important data. 2024-04-22T06:15:18.331 INFO:tasks.ceph.mon.a.smithi028.stderr: 2024-04-22T06:15:18.359 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.359+0000 7fd5df234000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:15:18.625 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.624+0000 7fd5df234000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:15:18.670 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:18.669+0000 7fd5df234000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:15:19.138 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T06:15:19.138 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:15:19.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T06:15:19.166 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-22T06:15:19.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T06:15:19.232 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:15:19.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T06:15:19.256 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-22T06:15:19.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-22T06:15:19.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 34da3fdc-f913-4940-9363-3d63c07ace2f 0 2024-04-22T06:15:19.382 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.382+0000 7fd5df234000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:15:19.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:19.461+0000 7fc391f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:19.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:19.470+0000 7fc391f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:19.475 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.474+0000 7fd5df234000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:15:19.519 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.518+0000 7fd5df234000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:15:19.551 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-04-22T06:15:19.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f 1 2024-04-22T06:15:19.692 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.691+0000 7fd5df234000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:15:19.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:19.701+0000 7f10df7f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:19.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:19.710+0000 7f10df7f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:19.748 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.748+0000 7fd5df234000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:15:19.791 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-22T06:15:19.802 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 0c22b725-abb9-4d41-9032-d229972f8e4e 2 2024-04-22T06:15:19.803 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.802+0000 7fd5df234000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:15:19.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:19.903+0000 7f71d9c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:19.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:19.912+0000 7f71d9c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:19.924 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:19.923+0000 7fd5df234000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:15:19.992 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-04-22T06:15:20.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 839db4fd-ba13-4221-95b4-95ca5caf8f15 3 2024-04-22T06:15:20.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:20.105+0000 7fd10851d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:20.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-22T06:15:20.114+0000 7fd10851d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:20.193 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-04-22T06:15:20.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-22T06:15:20.240 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T06:15:20.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T06:15:20.282 INFO:tasks.ceph.osd.0:Started 2024-04-22T06:15:20.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-22T06:15:20.316 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T06:15:20.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T06:15:20.317 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:20.315+0000 7fd5df234000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:15:20.359 INFO:tasks.ceph.osd.1:Started 2024-04-22T06:15:20.359 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-22T06:15:20.389 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T06:15:20.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T06:15:20.431 INFO:tasks.ceph.osd.2:Started 2024-04-22T06:15:20.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-22T06:15:20.460 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T06:15:20.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-22T06:15:20.502 INFO:tasks.ceph.osd.3:Started 2024-04-22T06:15:20.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:15:20.596 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:20.595+0000 7fd5df234000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:15:20.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:20.617+0000 7f5adc19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:20.626+0000 7f5adc19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:20.655 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:20.654+0000 7fd5df234000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:15:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:20.718 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"fsid":"507cb479-0568-4196-be48-f7b4ebc16539","created":"2024-04-22T06:15:18.034492+0000","modified":"2024-04-22T06:15:20.194137+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T06:15:20.194137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"34da3fdc-f913-4940-9363-3d63c07ace2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c22b725-abb9-4d41-9032-d229972f8e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"839db4fd-ba13-4221-95b4-95ca5caf8f15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:15:20.729 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T06:15:20.730 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T06:15:20.781 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:20.781+0000 7fd5df234000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:15:21.075 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.074+0000 7fd5df234000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:15:21.127 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.126+0000 7fd5df234000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:15:21.132 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T06:15:21.240 INFO:teuthology.misc.health.smithi028.stderr:2024-04-22T06:15:21.236+0000 7f96ba15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:21.245 INFO:teuthology.misc.health.smithi028.stderr:2024-04-22T06:15:21.245+0000 7f96ba15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:21.297 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.297+0000 7fd5df234000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:15:21.316 INFO:teuthology.misc.health.smithi028.stdout: 2024-04-22T06:15:21.316 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":7,"fsid":"507cb479-0568-4196-be48-f7b4ebc16539","created":"2024-04-22T06:15:18.034492+0000","modified":"2024-04-22T06:15:20.194137+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T06:15:20.194137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"34da3fdc-f913-4940-9363-3d63c07ace2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c22b725-abb9-4d41-9032-d229972f8e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"839db4fd-ba13-4221-95b4-95ca5caf8f15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:15:21.327 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-22T06:15:21.438 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.437+0000 7fd5df234000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:15:21.492 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.491+0000 7fd5df234000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:15:21.566 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.565+0000 7fd5df234000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:15:21.613 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:21.613+0000 7fd5df234000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:15:22.110 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:22.110+0000 7fd5df234000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:15:22.242 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:22.241+0000 7fd5df234000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:15:23.251 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:15:23.250+0000 7fd57e054640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-22T06:15:27.731 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T06:15:27.849 INFO:teuthology.misc.health.smithi028.stderr:2024-04-22T06:15:27.845+0000 7f8e716ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:27.854 INFO:teuthology.misc.health.smithi028.stderr:2024-04-22T06:15:27.854+0000 7f8e716ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:28.052 INFO:teuthology.misc.health.smithi028.stdout: 2024-04-22T06:15:28.052 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"507cb479-0568-4196-be48-f7b4ebc16539","created":"2024-04-22T06:15:18.034492+0000","modified":"2024-04-22T06:15:27.257222+0000","last_up_change":"2024-04-22T06:15:25.251684+0000","last_in_change":"2024-04-22T06:15:20.194137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:15:26.267082+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34da3fdc-f913-4940-9363-3d63c07ace2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1091203099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1091203099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6803","nonce":1091203099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1091203099}]},"public_addr":"172.21.15.28:6805/1091203099","cluster_addr":"172.21.15.28:6804/1091203099","heartbeat_back_addr":"172.21.15.28:6803/1091203099","heartbeat_front_addr":"172.21.15.28:6802/1091203099","state":["exists","up"]},{"osd":1,"uuid":"b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":979599584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":979599584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":979599584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":979599584}]},"public_addr":"172.21.15.28:6801/979599584","cluster_addr":"172.21.15.28:6800/979599584","heartbeat_back_addr":"172.21.15.28:6808/979599584","heartbeat_front_addr":"172.21.15.28:6807/979599584","state":["exists","up"]},{"osd":2,"uuid":"0c22b725-abb9-4d41-9032-d229972f8e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":78895659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":78895659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":78895659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":78895659}]},"public_addr":"172.21.15.103:6804/78895659","cluster_addr":"172.21.15.103:6805/78895659","heartbeat_back_addr":"172.21.15.103:6806/78895659","heartbeat_front_addr":"172.21.15.103:6807/78895659","state":["exists","up"]},{"osd":3,"uuid":"839db4fd-ba13-4221-95b4-95ca5caf8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":4235904975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":4235904975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":4235904975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4235904975}]},"public_addr":"172.21.15.103:6811/4235904975","cluster_addr":"172.21.15.103:6810/4235904975","heartbeat_back_addr":"172.21.15.103:6803/4235904975","heartbeat_front_addr":"172.21.15.103:6802/4235904975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:15:28.064 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-22T06:15:28.064 INFO:tasks.ceph:Creating RBD pool 2024-04-22T06:15:28.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T06:15:28.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:28.169+0000 7f2828e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:28.181+0000 7f2828e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'rbd' created 2024-04-22T06:15:29.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T06:15:29.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:29.447+0000 7fd8741ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:29.461+0000 7fd8741ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T06:15:30.342 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T06:15:30.342 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T06:15:30.342 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T06:15:30.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T06:15:30.343 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T06:15:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:30.491+0000 7f3ac935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:30.499+0000 7f3ac935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:30.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:30.764 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1380945539},{"type":"v1","addr":"172.21.15.103:6801","nonce":1380945539}]},"active_addr":"172.21.15.103:6801/1380945539","active_change":"2024-04-22T06:15:22.244499+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2391960782}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3575427978}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3805643199}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3545910528}]}]} 2024-04-22T06:15:30.769 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T06:15:30.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T06:15:30.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:15:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:30.877+0000 7fb9cf19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:30.885+0000 7fb9cf19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:31.084 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"fsid":"507cb479-0568-4196-be48-f7b4ebc16539","created":"2024-04-22T06:15:18.034492+0000","modified":"2024-04-22T06:15:30.315344+0000","last_up_change":"2024-04-22T06:15:25.251684+0000","last_in_change":"2024-04-22T06:15:20.194137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:15:26.267082+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T06:15:28.386228+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34da3fdc-f913-4940-9363-3d63c07ace2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1091203099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1091203099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6803","nonce":1091203099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1091203099}]},"public_addr":"172.21.15.28:6805/1091203099","cluster_addr":"172.21.15.28:6804/1091203099","heartbeat_back_addr":"172.21.15.28:6803/1091203099","heartbeat_front_addr":"172.21.15.28:6802/1091203099","state":["exists","up"]},{"osd":1,"uuid":"b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":979599584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":979599584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":979599584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":979599584}]},"public_addr":"172.21.15.28:6801/979599584","cluster_addr":"172.21.15.28:6800/979599584","heartbeat_back_addr":"172.21.15.28:6808/979599584","heartbeat_front_addr":"172.21.15.28:6807/979599584","state":["exists","up"]},{"osd":2,"uuid":"0c22b725-abb9-4d41-9032-d229972f8e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":78895659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":78895659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":78895659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":78895659}]},"public_addr":"172.21.15.103:6804/78895659","cluster_addr":"172.21.15.103:6805/78895659","heartbeat_back_addr":"172.21.15.103:6806/78895659","heartbeat_front_addr":"172.21.15.103:6807/78895659","state":["exists","up"]},{"osd":3,"uuid":"839db4fd-ba13-4221-95b4-95ca5caf8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":4235904975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":4235904975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":4235904975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4235904975}]},"public_addr":"172.21.15.103:6811/4235904975","cluster_addr":"172.21.15.103:6810/4235904975","heartbeat_back_addr":"172.21.15.103:6803/4235904975","heartbeat_front_addr":"172.21.15.103:6802/4235904975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:15:31.097 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T06:15:31.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:15:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.206+0000 7f380c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.215+0000 7f380c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.418 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:31.419 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":14,"fsid":"507cb479-0568-4196-be48-f7b4ebc16539","created":"2024-04-22T06:15:18.034492+0000","modified":"2024-04-22T06:15:31.318790+0000","last_up_change":"2024-04-22T06:15:25.251684+0000","last_in_change":"2024-04-22T06:15:20.194137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:15:26.267082+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T06:15:28.386228+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34da3fdc-f913-4940-9363-3d63c07ace2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1091203099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1091203099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6803","nonce":1091203099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1091203099}]},"public_addr":"172.21.15.28:6805/1091203099","cluster_addr":"172.21.15.28:6804/1091203099","heartbeat_back_addr":"172.21.15.28:6803/1091203099","heartbeat_front_addr":"172.21.15.28:6802/1091203099","state":["exists","up"]},{"osd":1,"uuid":"b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":979599584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":979599584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":979599584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":979599584}]},"public_addr":"172.21.15.28:6801/979599584","cluster_addr":"172.21.15.28:6800/979599584","heartbeat_back_addr":"172.21.15.28:6808/979599584","heartbeat_front_addr":"172.21.15.28:6807/979599584","state":["exists","up"]},{"osd":2,"uuid":"0c22b725-abb9-4d41-9032-d229972f8e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":78895659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":78895659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":78895659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":78895659}]},"public_addr":"172.21.15.103:6804/78895659","cluster_addr":"172.21.15.103:6805/78895659","heartbeat_back_addr":"172.21.15.103:6806/78895659","heartbeat_front_addr":"172.21.15.103:6807/78895659","state":["exists","up"]},{"osd":3,"uuid":"839db4fd-ba13-4221-95b4-95ca5caf8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":4235904975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":4235904975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":4235904975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4235904975}]},"public_addr":"172.21.15.103:6811/4235904975","cluster_addr":"172.21.15.103:6810/4235904975","heartbeat_back_addr":"172.21.15.103:6803/4235904975","heartbeat_front_addr":"172.21.15.103:6802/4235904975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:15:31.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T06:15:31.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T06:15:31.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T06:15:31.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-22T06:15:31.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.552+0000 7ff41ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.552+0000 7f7a65e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.552+0000 7ff17315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.552+0000 7fd504b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.563+0000 7ff41ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.563+0000 7ff17315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.564+0000 7f7a65e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.564+0000 7fd504b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.598 INFO:teuthology.orchestra.run.smithi028.stdout:38654705670 2024-04-22T06:15:31.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T06:15:31.602 INFO:teuthology.orchestra.run.smithi028.stdout:38654705670 2024-04-22T06:15:31.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:15:31.606 INFO:teuthology.orchestra.run.smithi028.stdout:38654705670 2024-04-22T06:15:31.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:15:31.608 INFO:teuthology.orchestra.run.smithi028.stdout:38654705670 2024-04-22T06:15:31.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:15:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.711+0000 7fda9346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.722+0000 7fda9346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.762+0000 7fd0e4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.762+0000 7f9864ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.764+0000 7f787946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.774+0000 7f9864ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.774+0000 7fd0e4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:31.775+0000 7f787946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:31.932 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:31.945 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-22T06:15:31.976 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:31.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-22T06:15:31.999 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:32.005 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:32.013 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-22T06:15:32.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-22T06:15:32.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T06:15:32.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:15:33.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:15:33.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:15:33.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.073+0000 7f7c6e6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.085+0000 7f7c6e6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.113+0000 7f3ebe42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.126+0000 7f3ebe42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.156+0000 7ff05f006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.157+0000 7fc0a606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.167+0000 7ff05f006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:33.167+0000 7fc0a606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:33.299 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:33.312 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-22T06:15:33.337 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:33.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-22T06:15:33.371 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:33.378 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:33.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-22T06:15:33.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-22T06:15:34.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T06:15:34.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:15:34.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:15:34.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:15:34.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.443+0000 7f116c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.456+0000 7f116c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.497+0000 7fcac755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.509+0000 7fcac755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.532+0000 7f91408ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.542+0000 7f91408ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.538+0000 7fc7a986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:34.550+0000 7fc7a986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:34.688 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:34.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-22T06:15:34.721 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:34.734 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-22T06:15:34.753 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:34.764 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:34.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-22T06:15:34.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-22T06:15:35.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T06:15:35.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:15:35.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:15:35.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:15:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.827+0000 7fc8e4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.840+0000 7fc8e4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.868+0000 7fd01175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.882+0000 7fd01175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.908+0000 7f9eae66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.908+0000 7facf0cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.918+0000 7facf0cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:35.918+0000 7f9eae66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:36.049 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:36.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-22T06:15:36.088 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:36.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-22T06:15:36.139 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:36.142 INFO:teuthology.orchestra.run.smithi028.stdout:38654705669 2024-04-22T06:15:36.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-22T06:15:36.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-22T06:15:37.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T06:15:37.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:15:37.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:15:37.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:15:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.189+0000 7f1a3bf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.202+0000 7f1a3bf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.228+0000 7f362a257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.240+0000 7f362a257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.274+0000 7fc76cced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.278+0000 7f8ba4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.284+0000 7fc76cced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.290+0000 7f8ba4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.416 INFO:teuthology.orchestra.run.smithi028.stdout:38654705674 2024-04-22T06:15:37.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.3 2024-04-22T06:15:37.430 DEBUG:teuthology.parallel:result is None 2024-04-22T06:15:37.454 INFO:teuthology.orchestra.run.smithi028.stdout:38654705674 2024-04-22T06:15:37.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.1 2024-04-22T06:15:37.468 DEBUG:teuthology.parallel:result is None 2024-04-22T06:15:37.487 INFO:teuthology.orchestra.run.smithi028.stdout:38654705674 2024-04-22T06:15:37.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.2 2024-04-22T06:15:37.501 DEBUG:teuthology.parallel:result is None 2024-04-22T06:15:37.503 INFO:teuthology.orchestra.run.smithi028.stdout:38654705674 2024-04-22T06:15:37.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.0 2024-04-22T06:15:37.516 DEBUG:teuthology.parallel:result is None 2024-04-22T06:15:37.516 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T06:15:37.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:15:37.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.666+0000 7f9bfe19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.674+0000 7f9bfe19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:37.871 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:15:37.883 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-22T06:15:36.250167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.971528"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.458673+0000","last_change":"2024-04-22T06:15:30.460326+0000","last_active":"2024-04-22T06:15:30.458673+0000","last_peered":"2024-04-22T06:15:30.458673+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.452351+0000","last_became_peered":"2024-04-22T06:15:30.452351+0000","last_unstale":"2024-04-22T06:15:30.458673+0000","last_undegraded":"2024-04-22T06:15:30.458673+0000","last_fullsized":"2024-04-22T06:15:30.458673+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.417023+0000","last_change":"2024-04-22T06:15:30.418704+0000","last_active":"2024-04-22T06:15:30.417023+0000","last_peered":"2024-04-22T06:15:30.417023+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.412641+0000","last_became_peered":"2024-04-22T06:15:30.412641+0000","last_unstale":"2024-04-22T06:15:30.417023+0000","last_undegraded":"2024-04-22T06:15:30.417023+0000","last_fullsized":"2024-04-22T06:15:30.417023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.339819+0000","last_change":"2024-04-22T06:15:30.408443+0000","last_active":"2024-04-22T06:15:31.339819+0000","last_peered":"2024-04-22T06:15:31.339819+0000","last_clean":"2024-04-22T06:15:31.339819+0000","last_became_active":"2024-04-22T06:15:30.403932+0000","last_became_peered":"2024-04-22T06:15:30.403932+0000","last_unstale":"2024-04-22T06:15:31.339819+0000","last_undegraded":"2024-04-22T06:15:31.339819+0000","last_fullsized":"2024-04-22T06:15:31.339819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.409670+0000","last_change":"2024-04-22T06:15:30.410515+0000","last_active":"2024-04-22T06:15:30.409670+0000","last_peered":"2024-04-22T06:15:30.409670+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.406142+0000","last_became_peered":"2024-04-22T06:15:30.406142+0000","last_unstale":"2024-04-22T06:15:30.409670+0000","last_undegraded":"2024-04-22T06:15:30.409670+0000","last_fullsized":"2024-04-22T06:15:30.409670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.423227+0000","last_change":"2024-04-22T06:15:30.424826+0000","last_active":"2024-04-22T06:15:30.423227+0000","last_peered":"2024-04-22T06:15:30.423227+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.416975+0000","last_became_peered":"2024-04-22T06:15:30.416975+0000","last_unstale":"2024-04-22T06:15:30.423227+0000","last_undegraded":"2024-04-22T06:15:30.423227+0000","last_fullsized":"2024-04-22T06:15:30.423227+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.403751+0000","last_change":"2024-04-22T06:15:30.405918+0000","last_active":"2024-04-22T06:15:30.403751+0000","last_peered":"2024-04-22T06:15:30.403751+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.397935+0000","last_became_peered":"2024-04-22T06:15:30.397935+0000","last_unstale":"2024-04-22T06:15:30.403751+0000","last_undegraded":"2024-04-22T06:15:30.403751+0000","last_fullsized":"2024-04-22T06:15:30.403751+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.428277+0000","last_change":"2024-04-22T06:15:30.429530+0000","last_active":"2024-04-22T06:15:30.428277+0000","last_peered":"2024-04-22T06:15:30.428277+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.424410+0000","last_became_peered":"2024-04-22T06:15:30.424410+0000","last_unstale":"2024-04-22T06:15:30.428277+0000","last_undegraded":"2024-04-22T06:15:30.428277+0000","last_fullsized":"2024-04-22T06:15:30.428277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.444364+0000","last_change":"2024-04-22T06:15:30.446240+0000","last_active":"2024-04-22T06:15:30.444364+0000","last_peered":"2024-04-22T06:15:30.444364+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.439253+0000","last_became_peered":"2024-04-22T06:15:30.439253+0000","last_unstale":"2024-04-22T06:15:30.444364+0000","last_undegraded":"2024-04-22T06:15:30.444364+0000","last_fullsized":"2024-04-22T06:15:30.444364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.335083+0000","last_change":"2024-04-22T06:15:28.316477+0000","last_active":"2024-04-22T06:15:31.335083+0000","last_peered":"2024-04-22T06:15:31.335083+0000","last_clean":"2024-04-22T06:15:31.335083+0000","last_became_active":"2024-04-22T06:15:28.310427+0000","last_became_peered":"2024-04-22T06:15:28.310427+0000","last_unstale":"2024-04-22T06:15:31.335083+0000","last_undegraded":"2024-04-22T06:15:31.335083+0000","last_fullsized":"2024-04-22T06:15:31.335083+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:15:37.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:15:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:37.992+0000 7f542959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:38.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:38.001+0000 7f542959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:38.199 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:15:38.212 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-22T06:15:36.250167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.971528"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.458673+0000","last_change":"2024-04-22T06:15:30.460326+0000","last_active":"2024-04-22T06:15:30.458673+0000","last_peered":"2024-04-22T06:15:30.458673+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.452351+0000","last_became_peered":"2024-04-22T06:15:30.452351+0000","last_unstale":"2024-04-22T06:15:30.458673+0000","last_undegraded":"2024-04-22T06:15:30.458673+0000","last_fullsized":"2024-04-22T06:15:30.458673+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.417023+0000","last_change":"2024-04-22T06:15:30.418704+0000","last_active":"2024-04-22T06:15:30.417023+0000","last_peered":"2024-04-22T06:15:30.417023+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.412641+0000","last_became_peered":"2024-04-22T06:15:30.412641+0000","last_unstale":"2024-04-22T06:15:30.417023+0000","last_undegraded":"2024-04-22T06:15:30.417023+0000","last_fullsized":"2024-04-22T06:15:30.417023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.339819+0000","last_change":"2024-04-22T06:15:30.408443+0000","last_active":"2024-04-22T06:15:31.339819+0000","last_peered":"2024-04-22T06:15:31.339819+0000","last_clean":"2024-04-22T06:15:31.339819+0000","last_became_active":"2024-04-22T06:15:30.403932+0000","last_became_peered":"2024-04-22T06:15:30.403932+0000","last_unstale":"2024-04-22T06:15:31.339819+0000","last_undegraded":"2024-04-22T06:15:31.339819+0000","last_fullsized":"2024-04-22T06:15:31.339819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.409670+0000","last_change":"2024-04-22T06:15:30.410515+0000","last_active":"2024-04-22T06:15:30.409670+0000","last_peered":"2024-04-22T06:15:30.409670+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.406142+0000","last_became_peered":"2024-04-22T06:15:30.406142+0000","last_unstale":"2024-04-22T06:15:30.409670+0000","last_undegraded":"2024-04-22T06:15:30.409670+0000","last_fullsized":"2024-04-22T06:15:30.409670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.423227+0000","last_change":"2024-04-22T06:15:30.424826+0000","last_active":"2024-04-22T06:15:30.423227+0000","last_peered":"2024-04-22T06:15:30.423227+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.416975+0000","last_became_peered":"2024-04-22T06:15:30.416975+0000","last_unstale":"2024-04-22T06:15:30.423227+0000","last_undegraded":"2024-04-22T06:15:30.423227+0000","last_fullsized":"2024-04-22T06:15:30.423227+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.403751+0000","last_change":"2024-04-22T06:15:30.405918+0000","last_active":"2024-04-22T06:15:30.403751+0000","last_peered":"2024-04-22T06:15:30.403751+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.397935+0000","last_became_peered":"2024-04-22T06:15:30.397935+0000","last_unstale":"2024-04-22T06:15:30.403751+0000","last_undegraded":"2024-04-22T06:15:30.403751+0000","last_fullsized":"2024-04-22T06:15:30.403751+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.428277+0000","last_change":"2024-04-22T06:15:30.429530+0000","last_active":"2024-04-22T06:15:30.428277+0000","last_peered":"2024-04-22T06:15:30.428277+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.424410+0000","last_became_peered":"2024-04-22T06:15:30.424410+0000","last_unstale":"2024-04-22T06:15:30.428277+0000","last_undegraded":"2024-04-22T06:15:30.428277+0000","last_fullsized":"2024-04-22T06:15:30.428277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.444364+0000","last_change":"2024-04-22T06:15:30.446240+0000","last_active":"2024-04-22T06:15:30.444364+0000","last_peered":"2024-04-22T06:15:30.444364+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.439253+0000","last_became_peered":"2024-04-22T06:15:30.439253+0000","last_unstale":"2024-04-22T06:15:30.444364+0000","last_undegraded":"2024-04-22T06:15:30.444364+0000","last_fullsized":"2024-04-22T06:15:30.444364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.335083+0000","last_change":"2024-04-22T06:15:28.316477+0000","last_active":"2024-04-22T06:15:31.335083+0000","last_peered":"2024-04-22T06:15:31.335083+0000","last_clean":"2024-04-22T06:15:31.335083+0000","last_became_active":"2024-04-22T06:15:28.310427+0000","last_became_peered":"2024-04-22T06:15:28.310427+0000","last_unstale":"2024-04-22T06:15:31.335083+0000","last_undegraded":"2024-04-22T06:15:31.335083+0000","last_fullsized":"2024-04-22T06:15:31.335083+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:15:38.212 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T06:15:38.213 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T06:15:38.213 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T06:15:38.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T06:15:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:38.321+0000 7fc09bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:38.328+0000 7fc09bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:38.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T06:15:38.593 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T06:15:38.594 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-22T06:15:38.629 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-22T06:15:38.630 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}} 2024-04-22T06:15:38.630 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-22T06:15:38.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-22T06:15:38.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:38.739+0000 7f5307f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:38.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:38.751+0000 7f5307f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:38.967 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-22T06:15:38.967 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T06:15:38.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:15:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:39.074+0000 7f5b5a224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:39.083+0000 7f5b5a224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:39.277 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:15:39.290 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-22T06:15:38.250738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.972099"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.458673+0000","last_change":"2024-04-22T06:15:30.460326+0000","last_active":"2024-04-22T06:15:30.458673+0000","last_peered":"2024-04-22T06:15:30.458673+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.452351+0000","last_became_peered":"2024-04-22T06:15:30.452351+0000","last_unstale":"2024-04-22T06:15:30.458673+0000","last_undegraded":"2024-04-22T06:15:30.458673+0000","last_fullsized":"2024-04-22T06:15:30.458673+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.417023+0000","last_change":"2024-04-22T06:15:30.418704+0000","last_active":"2024-04-22T06:15:30.417023+0000","last_peered":"2024-04-22T06:15:30.417023+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.412641+0000","last_became_peered":"2024-04-22T06:15:30.412641+0000","last_unstale":"2024-04-22T06:15:30.417023+0000","last_undegraded":"2024-04-22T06:15:30.417023+0000","last_fullsized":"2024-04-22T06:15:30.417023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.339819+0000","last_change":"2024-04-22T06:15:30.408443+0000","last_active":"2024-04-22T06:15:31.339819+0000","last_peered":"2024-04-22T06:15:31.339819+0000","last_clean":"2024-04-22T06:15:31.339819+0000","last_became_active":"2024-04-22T06:15:30.403932+0000","last_became_peered":"2024-04-22T06:15:30.403932+0000","last_unstale":"2024-04-22T06:15:31.339819+0000","last_undegraded":"2024-04-22T06:15:31.339819+0000","last_fullsized":"2024-04-22T06:15:31.339819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.409670+0000","last_change":"2024-04-22T06:15:30.410515+0000","last_active":"2024-04-22T06:15:30.409670+0000","last_peered":"2024-04-22T06:15:30.409670+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.406142+0000","last_became_peered":"2024-04-22T06:15:30.406142+0000","last_unstale":"2024-04-22T06:15:30.409670+0000","last_undegraded":"2024-04-22T06:15:30.409670+0000","last_fullsized":"2024-04-22T06:15:30.409670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.423227+0000","last_change":"2024-04-22T06:15:30.424826+0000","last_active":"2024-04-22T06:15:30.423227+0000","last_peered":"2024-04-22T06:15:30.423227+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.416975+0000","last_became_peered":"2024-04-22T06:15:30.416975+0000","last_unstale":"2024-04-22T06:15:30.423227+0000","last_undegraded":"2024-04-22T06:15:30.423227+0000","last_fullsized":"2024-04-22T06:15:30.423227+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.403751+0000","last_change":"2024-04-22T06:15:30.405918+0000","last_active":"2024-04-22T06:15:30.403751+0000","last_peered":"2024-04-22T06:15:30.403751+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.397935+0000","last_became_peered":"2024-04-22T06:15:30.397935+0000","last_unstale":"2024-04-22T06:15:30.403751+0000","last_undegraded":"2024-04-22T06:15:30.403751+0000","last_fullsized":"2024-04-22T06:15:30.403751+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.428277+0000","last_change":"2024-04-22T06:15:30.429530+0000","last_active":"2024-04-22T06:15:30.428277+0000","last_peered":"2024-04-22T06:15:30.428277+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.424410+0000","last_became_peered":"2024-04-22T06:15:30.424410+0000","last_unstale":"2024-04-22T06:15:30.428277+0000","last_undegraded":"2024-04-22T06:15:30.428277+0000","last_fullsized":"2024-04-22T06:15:30.428277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.444364+0000","last_change":"2024-04-22T06:15:30.446240+0000","last_active":"2024-04-22T06:15:30.444364+0000","last_peered":"2024-04-22T06:15:30.444364+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.439253+0000","last_became_peered":"2024-04-22T06:15:30.439253+0000","last_unstale":"2024-04-22T06:15:30.444364+0000","last_undegraded":"2024-04-22T06:15:30.444364+0000","last_fullsized":"2024-04-22T06:15:30.444364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.335083+0000","last_change":"2024-04-22T06:15:28.316477+0000","last_active":"2024-04-22T06:15:31.335083+0000","last_peered":"2024-04-22T06:15:31.335083+0000","last_clean":"2024-04-22T06:15:31.335083+0000","last_became_active":"2024-04-22T06:15:28.310427+0000","last_became_peered":"2024-04-22T06:15:28.310427+0000","last_unstale":"2024-04-22T06:15:31.335083+0000","last_undegraded":"2024-04-22T06:15:31.335083+0000","last_fullsized":"2024-04-22T06:15:31.335083+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:15:39.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:15:39.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:39.399+0000 7f336aa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:39.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:39.407+0000 7f336aa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:39.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:15:39.615 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-22T06:15:38.250738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.972099"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.458673+0000","last_change":"2024-04-22T06:15:30.460326+0000","last_active":"2024-04-22T06:15:30.458673+0000","last_peered":"2024-04-22T06:15:30.458673+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.452351+0000","last_became_peered":"2024-04-22T06:15:30.452351+0000","last_unstale":"2024-04-22T06:15:30.458673+0000","last_undegraded":"2024-04-22T06:15:30.458673+0000","last_fullsized":"2024-04-22T06:15:30.458673+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.417023+0000","last_change":"2024-04-22T06:15:30.418704+0000","last_active":"2024-04-22T06:15:30.417023+0000","last_peered":"2024-04-22T06:15:30.417023+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.412641+0000","last_became_peered":"2024-04-22T06:15:30.412641+0000","last_unstale":"2024-04-22T06:15:30.417023+0000","last_undegraded":"2024-04-22T06:15:30.417023+0000","last_fullsized":"2024-04-22T06:15:30.417023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.339819+0000","last_change":"2024-04-22T06:15:30.408443+0000","last_active":"2024-04-22T06:15:31.339819+0000","last_peered":"2024-04-22T06:15:31.339819+0000","last_clean":"2024-04-22T06:15:31.339819+0000","last_became_active":"2024-04-22T06:15:30.403932+0000","last_became_peered":"2024-04-22T06:15:30.403932+0000","last_unstale":"2024-04-22T06:15:31.339819+0000","last_undegraded":"2024-04-22T06:15:31.339819+0000","last_fullsized":"2024-04-22T06:15:31.339819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.409670+0000","last_change":"2024-04-22T06:15:30.410515+0000","last_active":"2024-04-22T06:15:30.409670+0000","last_peered":"2024-04-22T06:15:30.409670+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.406142+0000","last_became_peered":"2024-04-22T06:15:30.406142+0000","last_unstale":"2024-04-22T06:15:30.409670+0000","last_undegraded":"2024-04-22T06:15:30.409670+0000","last_fullsized":"2024-04-22T06:15:30.409670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.423227+0000","last_change":"2024-04-22T06:15:30.424826+0000","last_active":"2024-04-22T06:15:30.423227+0000","last_peered":"2024-04-22T06:15:30.423227+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.416975+0000","last_became_peered":"2024-04-22T06:15:30.416975+0000","last_unstale":"2024-04-22T06:15:30.423227+0000","last_undegraded":"2024-04-22T06:15:30.423227+0000","last_fullsized":"2024-04-22T06:15:30.423227+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.403751+0000","last_change":"2024-04-22T06:15:30.405918+0000","last_active":"2024-04-22T06:15:30.403751+0000","last_peered":"2024-04-22T06:15:30.403751+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.397935+0000","last_became_peered":"2024-04-22T06:15:30.397935+0000","last_unstale":"2024-04-22T06:15:30.403751+0000","last_undegraded":"2024-04-22T06:15:30.403751+0000","last_fullsized":"2024-04-22T06:15:30.403751+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.428277+0000","last_change":"2024-04-22T06:15:30.429530+0000","last_active":"2024-04-22T06:15:30.428277+0000","last_peered":"2024-04-22T06:15:30.428277+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.424410+0000","last_became_peered":"2024-04-22T06:15:30.424410+0000","last_unstale":"2024-04-22T06:15:30.428277+0000","last_undegraded":"2024-04-22T06:15:30.428277+0000","last_fullsized":"2024-04-22T06:15:30.428277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T06:15:30.444364+0000","last_change":"2024-04-22T06:15:30.446240+0000","last_active":"2024-04-22T06:15:30.444364+0000","last_peered":"2024-04-22T06:15:30.444364+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"2024-04-22T06:15:30.439253+0000","last_became_peered":"2024-04-22T06:15:30.439253+0000","last_unstale":"2024-04-22T06:15:30.444364+0000","last_undegraded":"2024-04-22T06:15:30.444364+0000","last_fullsized":"2024-04-22T06:15:30.444364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T06:15:31.335083+0000","last_change":"2024-04-22T06:15:28.316477+0000","last_active":"2024-04-22T06:15:31.335083+0000","last_peered":"2024-04-22T06:15:31.335083+0000","last_clean":"2024-04-22T06:15:31.335083+0000","last_became_active":"2024-04-22T06:15:28.310427+0000","last_became_peered":"2024-04-22T06:15:28.310427+0000","last_unstale":"2024-04-22T06:15:31.335083+0000","last_undegraded":"2024-04-22T06:15:31.335083+0000","last_fullsized":"2024-04-22T06:15:31.335083+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:15:39.615 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T06:15:39.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-22T06:15:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:39.723+0000 7f2bc2ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:39.731+0000 7f2bc2ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 14 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 507cb479-0568-4196-be48-f7b4ebc16539 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-22T06:15:18.034492+0000 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-22T06:15:31.318790+0000 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 3 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-04-22T06:15:39.929 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client jewel 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:allow_crimson=true 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 4 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.28:6805/1091203099 v2:172.21.15.28:6804/1091203099 exists,up 34da3fdc-f913-4940-9363-3d63c07ace2f 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.28:6801/979599584 v2:172.21.15.28:6800/979599584 exists,up b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f 2024-04-22T06:15:39.930 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6804/78895659 v2:172.21.15.103:6805/78895659 exists,up 0c22b725-abb9-4d41-9032-d229972f8e4e 2024-04-22T06:15:39.931 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6811/4235904975 v2:172.21.15.103:6810/4235904975 exists,up 839db4fd-ba13-4221-95b4-95ca5caf8f15 2024-04-22T06:15:39.941 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.28:6805/1091203099 v2:172.21.15.28:6804/1091203099 exists,up 34da3fdc-f913-4940-9363-3d63c07ace2f', 'osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.28:6801/979599584 v2:172.21.15.28:6800/979599584 exists,up b7184cbd-8bd8-40e4-b9be-c8f0cc7b763f', 'osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6804/78895659 v2:172.21.15.103:6805/78895659 exists,up 0c22b725-abb9-4d41-9032-d229972f8e4e', 'osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6811/4235904975 v2:172.21.15.103:6810/4235904975 exists,up 839db4fd-ba13-4221-95b4-95ca5caf8f15'] 2024-04-22T06:15:39.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mon.a", 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster": "ceph", 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket_mode": "", 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-22T06:15:40.051 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_client_required": "cephx, none", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_cluster_required": "cephx", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_debug": "true", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_required": "cephx", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_supported": "cephx", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio": "true", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_poll_ms": "250", 2024-04-22T06:15:40.052 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_reap_max": "16", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_async_discard_threads": "0", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_block_size": "4096", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio": "false", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_inflight_ios": "false", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_enable_discard": "false", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry": "3", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash": "0", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring": "false", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_hipri": "false", 2024-04-22T06:15:40.053 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_type": "", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_alloc_size": "1048576", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_allocator": "hybrid", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_buffered_io": "true", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_for_zeros": "false", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_compact_log_sync": "false", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-22T06:15:40.054 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_log_runway": "4194304", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_prefetch": "1048576", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_flush_size": "524288", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_log_runway": "1048576", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery": "false", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_sync_write": "false", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocation_from_file": "true", 2024-04-22T06:15:40.055 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocator": "hybrid", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_create": "true", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_create": "false", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_path": "", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_size": "0", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_path": "", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_preallocate_file": "false", 2024-04-22T06:15:40.056 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_size": "107374182400", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_create": "false", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_path": "", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_size": "100663296", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs": "true", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune": "true", 2024-04-22T06:15:40.057 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size": "0", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_type": "2q", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_clone_cow": "true", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-22T06:15:40.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_mode": "none", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_csum_type": "crc32c", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_freelist": "false", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-22T06:15:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_small_allocations": "0", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_read": "true", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_write": "false", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-22T06:15:40.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fail_eio": "false", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-22T06:15:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-22T06:15:40.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_ignore_data_csum": "false", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_op_age": "5.000000", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_alloc_size": "0", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size": "0", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_deferred_txc": "32", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size": "0", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-22T06:15:40.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_nid_prealloc": "1024", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_retry_disk_reads": "3", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cf": "true", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-22T06:15:40.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_mem": "512", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_tracing": "false", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-22T06:15:40.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_zero_block_detection": "false", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-22T06:15:40.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_signatures": "false", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_version": "2", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_signatures": "false", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_version": "2", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_signatures": "false", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_version": "2", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_sign_messages": "true", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "chdir": "", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client_acl_type": "", 2024-04-22T06:15:40.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client_asio_thread_count": "2", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_mid": "0.750000", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_size": "16384", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_caps_release_delay": "5", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_check_pool_perm": "true", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_force_sync_read": "false", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_getattr_caps": "false", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_tick_delay": "0", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_remount": "false", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_dirsize_rbytes": "true", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_force_lazyio": "false", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_fs": "", 2024-04-22T06:15:40.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_release_failure": "false", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_inline_size": "4096", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mds_namespace": "", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_metadata": "", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_gid": "-1", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_timeout": "300", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_uid": "-1", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mountpoint": "/", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_notify_timeout": "10", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc": "true", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty": "104857600", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_objects": "1000", 2024-04-22T06:15:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_size": "209715200", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_target_dirty": "8388608", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_permissions": "true", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota": "true", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota_df": "true", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_bytes": "0", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_periods": "4", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_min": "131072", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_reconnect_stale": "false", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_shutdown_timeout": "30", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_snapdir": ".snap", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_tick_interval": "1", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_trace": "", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_try_dentry_invalidate": "false", 2024-04-22T06:15:40.070 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_faked_inos": "false", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_random_mds": "false", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog": "false", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_port": "12201", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_monitors": "default=true", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog": "false", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_level": "info", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "-", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network": "", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network_interface": "", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_isal": "false", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_level": "5", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_winsize": "-15", 2024-04-22T06:15:40.071 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zstd_level": "1", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_op_num_threads": "6", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_stat_interval": "0", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_seastar_cpu_cores": "", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook": "", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook_timeout": "10", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "daemonize": "false", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "daos_pool": "tank", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-22T06:15:40.072 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_alienstore": "0/5", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_allow_any_pool_priority": "false", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok": "1/5", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok_assert_abort": "false", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asserts_on_shutdown": "true", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_auth": "20/20", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bdev": "1/3", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluefs": "1/5", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluestore": "1/5", 2024-04-22T06:15:40.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_buffer": "0/1", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ceph_exporter": "1/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephfs_mirror": "0/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephsqlite": "0/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_client": "0/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_compressor": "1/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_context": "0/1", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crimson_interrupt": "0/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crush": "1/1", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crypto": "1/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cyanstore": "0/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_deliberately_leak_memory": "false", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_disable_randomized_ping": "false", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_dpdk": "1/5", 2024-04-22T06:15:40.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_eventtrace": "1/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filer": "0/1", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filestore": "1/3", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_finisher": "1/1", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_fuse": "1/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeat_testing_span": "0", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeatmap": "1/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_javaclient": "1/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journal": "1/3", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journaler": "0/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_kstore": "1/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_lockdep": "0/1", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mclock": "1/5", 2024-04-22T06:15:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_balancer": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_locker": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log_expire": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_migrator": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_quiesce": "3/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_memstore": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgr": "2/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgrc": "1/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mon": "20/20", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_monc": "0/10", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ms": "1/1", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_none": "0/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objclass": "0/5", 2024-04-22T06:15:40.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objectcacher": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objecter": "0/1", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_optracker": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_osd": "1/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_paxos": "20/20", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_perfcounter": "1/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_prioritycache": "1/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rados": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_mirror": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_pwl": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_replay": "0/5", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_refs": "0/0", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_reserver": "1/1", 2024-04-22T06:15:40.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_access": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_datacache": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_dbstore": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_flight": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_notification": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_sync": "1/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rocksdb": "4/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore": "0/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_backref": "0/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cache": "0/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cleaner": "0/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_device": "0/5", 2024-04-22T06:15:40.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_epm": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_journal": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_lba": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_odata": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_omap": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_onode": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_t": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_tm": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_striper": "0/1", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_test": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_throttle": "1/1", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_timer": "0/1", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_tp": "0/5", 2024-04-22T06:15:40.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_trace": "1/5", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "device_failure_prediction_mode": "none", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_graylog": "false", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_journald": "false", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_stderr": "true", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_syslog": "false", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event_tracing": "false", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_addr": "0.0.0.0", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_http_port": "9926", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_prio_limit": "5", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-22T06:15:40.080 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sort_metrics": "true", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_stats_period": "5", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "fake_statfs_for_testing": "0", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "fatal_signal_handlers": "true", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_purge_ops": "10", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_truncate_ops": "128", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_apply_finisher_threads": "1", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_blackhole": "false", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_clone_range": "true", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_snap": "true", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_caller_concurrency": "10", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_collect_device_partition_information": "true", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_commit_timeout": "600.000000", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_inject_read_err": "false", 2024-04-22T06:15:40.081 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_omap_check": "false", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_verify_split": "false", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_dump_file": "", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fadvise": "true", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fail_eio": "true", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_shards": "16", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_size": "128", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap": "false", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap_threshold": "4096", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_inject_stall": "0", 2024-04-22T06:15:40.082 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_parallel": "false", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_trailing": "false", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_writeahead": "false", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_kill_at": "0", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs": "0", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-22T06:15:40.083 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size": "0", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_merge_threshold": "-10", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_odsync_write": "false", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend": "rocksdb", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend_path": "", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_timeout": "60", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_threads": "2", 2024-04-22T06:15:40.084 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_punch_hole": "false", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_ops": "50", 2024-04-22T06:15:40.085 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_seek_data_hole": "false", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc": "false", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_splice": "false", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_multiple": "2", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_rand_factor": "20", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_update_to": "1000", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-22T06:15:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_enable": "true", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_xfs_extsize": "false", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_zfs_snap": "false", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fio_dir": "/tmp/fio", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "507cb479-0568-4196-be48-f7b4ebc16539", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_allow_other": "true", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_atomic_o_trunc": "true", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_big_writes": "true", 2024-04-22T06:15:40.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_debug": "false", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_default_permissions": "false", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_disable_pagecache": "false", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_max_write": "0", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_multithreaded": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_require_active_mds": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_set_user_groups": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_move": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_read": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_write": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_use_invalidate_cb": "true", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_target_name": "ceph", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_file": "", 2024-04-22T06:15:40.088 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_inject_failure": "0", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_interval": "5", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "host": "smithi028", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-22T06:15:40.089 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "inject_early_sigterm": "false", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_agent_port": "6799", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_tracing_enable": "false", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_aio": "true", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_align_min_size": "65536", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_align": "true", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_size": "4096", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_dio": "true", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_discard": "false", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_force_aio": "false", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_ignore_corruption": "false", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_bytes": "10485760", 2024-04-22T06:15:40.090 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_entries": "100", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_replay_from": "0", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_write_header_frequency": "0", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_zero_on_create": "false", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prefetch_periods": "10", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prezero_periods": "5", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_write_head_interval": "15", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "keyfile": "", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_backend": "rocksdb", 2024-04-22T06:15:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_default_stripe_size": "65536", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount": "false", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_bytes": "67108864", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_ops": "512", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_nid_prealloc": "1024", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_onode_map_size": "1024", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_submit_transaction": "false", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_transaction": "false", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "librados_thread_count": "2", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep": "false", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep_force_backtrace": "false", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "log_coarse_timestamps": "true", 2024-04-22T06:15:40.092 INFO:teuthology.orchestra.run.smithi028.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_flush_on_exit": "false", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_host": "127.0.0.1", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_port": "12201", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_new": "1000", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_recent": "10000", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stderr_prefix": "", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stop_at_utilization": "0.970000", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": "true", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_graylog": "false", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_journald": "false", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_stderr": "false", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_syslog": "false", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max_rotating_auth_attempts": "10", 2024-04-22T06:15:40.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_action_on_write_error": "1", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_alternate_name_max": "8192", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_asio_thread_count": "2", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_export_pin": "true", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_dirs": "true", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_interval": "5", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_interval": "10", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max": "-1", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max_until": "-1", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_merge_size": "50", 2024-04-22T06:15:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_midchunk": "0.300000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_start": "0.200000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_minchunk": "0.001000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_mode": "0", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_max": "1.200000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_min": "0.800000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_overload_epochs": "2", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_bits": "3", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_size": "10000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-22T06:15:40.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_target_decay": "10.000000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_grace": "15.000000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_interval": "4.000000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_mid": "0.700000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_delay": "0", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_release_free_interval": "10", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_reservation": "0.050000", 2024-04-22T06:15:40.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_interval": "1", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_threshold": "262144", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_prealloc_inos": "1000", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_connect_bootstrapping": "false", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_damage_table_max_entries": "10000", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_auth_pins": "false", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_frag": "false", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_scatterstat": "false", 2024-04-22T06:15:40.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_subtrees": "false", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_decay_halflife": "5.000000", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_default_dir_hash": "2", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_defer_session_stale": "true", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_deny_all_reconnect": "false", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_keys_per_op": "16384", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_commit_size": "10", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_entries": "0", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_prefetch": "true", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_on_map": "false", 2024-04-22T06:15:40.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_early_reply": "true", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enable_op_tracker": "true", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enforce_unique_name": "true", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random": "true", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-22T06:15:40.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_summarize_threshold": "10", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_health_dummy": "false", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_migrator_session_race": "false", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_join_fs": "", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_journal_format": "1", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_create_at": "0", 2024-04-22T06:15:40.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_export_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_import_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_expire_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_replay_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_link_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_mdstable_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_openc_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_rename_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_shutdown_at": "0", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_event_large_threshold": "524288", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_events_per_segment": "1024", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-22T06:15:40.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_events": "-1", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_segments": "128", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_pause": "false", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_segment_size": "0", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_threshold": "128", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_warn_factor": "2.000000", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_caps_per_client": "1048576", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_flushes": "100000", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_requests": "100000", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_export_size": "20971520", 2024-04-22T06:15:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_file_recover": "32", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_files": "64", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops": "8192", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_snaps_per_dir": "100", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_metrics_update_interval": "2", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_per_client": "100", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_working_set": "10000", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_numa_node": "-1", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_complaint_time": "30.000000", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_duration": "600", 2024-04-22T06:15:40.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_size": "20", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_size": "20", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_log_threshold": "5", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_grace": "15", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_interval": "5", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_caps": "30000", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_threshold": "262144", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_interval": "1.000000", 2024-04-22T06:15:40.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_gid": "0", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_uid": "0", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-22T06:15:40.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_shutdown_check": "0", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_skip_ino": "0", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_max_uid": "4294967294", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_min_uid": "0", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_rstat": "false", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_standby_replay_damaged": "false", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_symlink_recovery": "true", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_exports": "0", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_fragments": "0", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_tick_interval": "5.000000", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_valgrind_exit": "false", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_backtrace": "1", 2024-04-22T06:15:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_scatter": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_sessions": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool_debug": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_device_bytes": "1073741824", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_set": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_size": "65536", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_bytes": "134217728", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_messages": "512", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-22T06:15:40.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_disabled_modules": "", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_max_pg_creating": "1024", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_max_pg_num_change": "128", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_bytes": "134217728", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_messages": "128", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_bytes": "134217728", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_messages": "128", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_bytes": "536870912", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_messages": "8192", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_pool": "true", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-22T06:15:40.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_standby_modules": "true", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_period": "5", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_threshold": "5", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_tick_period": "2", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_delete": "true", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_size_one": "true", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_auth_validate_all_caps": "true", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_bytes": "104857600", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_directed_command_retry": "2", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-22T06:15:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_parallel": "3", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_log_interval": "1.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_interval": "10.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_target_rank": "-1", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_level": "debug", 2024-04-22T06:15:40.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_file": "true", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_journald": "false", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_bootstrap": "false", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_start": "false", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_trim": "true", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-22T06:15:40.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cpu_threads": "4", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_crush_min_required_version": "hammer", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_daemon_bytes": "419430400", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_crit": "5", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_warn": "5", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_size_warn": "16106127360", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_json": "false", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_transactions": "false", 2024-04-22T06:15:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_extra_checks": "false", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_reef": "false", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_squid": "false", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_mkfs_grace": "120", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_uptime_grace": "60", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_default_strategy": "1", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_timeout": "5.000000", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_divisor": "2", 2024-04-22T06:15:40.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_enable_op_tracker": "true", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_fake_pool_delete": "false", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_force_quorum_join": "false", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_globalid_prealloc": "10000", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_detail_to_clog": "false", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_log_update_period": "5", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_max_detail": "50", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog": "true", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_interval": "600", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host": "172.21.15.28", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host_override": "", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_initial_members": "", 2024-04-22T06:15:40.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease": "5.000000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_full_interval": "50", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max": "10000", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max_summary": "50", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_epochs": "500", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-22T06:15:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_osd": "10000", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pg_per_osd": "10000", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pool_pg_num": "65536", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_force_trim_to": "0", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_skip_sanity": "false", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_autotune": "true", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_target": "2147483648", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_beacon_grace": "30", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_digest_period": "5", 2024-04-22T06:15:40.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_inactive_grace": "60", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_min_osdmap_epochs": "50", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_complaint_time": "30", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_duration": "600", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_size": "20", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_size": "20", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_log_threshold": "5", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_in": "false", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-22T06:15:40.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size": "500", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_interval": "600", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_force_trim_to": "0", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-22T06:15:40.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_down_reporters": "2", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-22T06:15:40.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_report_timeout": "900", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_stuck_threshold": "60", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-22T06:15:40.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_probe_timeout": "2.000000", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_change": "0.050000", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_osds": "4", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_interval": "86400", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_max_keys": "100", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_timeout": "300", 2024-04-22T06:15:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_session_timeout": "300", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_smart_report_timeout": "5", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stat_smooth_intervals": "6", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_min_size": "2", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_size": "4", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_debug": "false", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_provider_kill_at": "0", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_requester_kill_at": "0", 2024-04-22T06:15:40.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_timeout": "60.000000", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_target_pg_per_osd": "100", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_tick_interval": "5", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_interval": "300.000000", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_use_min_delay_socket": "false", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_older_version_delay": "604800", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-22T06:15:40.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_misplaced": "false", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_older_version": "true", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": "", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-22T06:15:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_tracing_enabled": "false", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_op_threads": "3", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_cm": "false", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_device_name": "", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_dscp": "96", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_local_gid": "", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-22T06:15:40.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_port_num": "1", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_sl": "3", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_support_srq": "true", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_type": "ib", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_reap_threshold": "5", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_before_connect": "false", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv4": "true", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv6": "false", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr1": "true", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr2": "true", 2024-04-22T06:15:40.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_max": "7568", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_min": "6800", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_count": "3", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_delay": "5", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_client": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mds": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mgr": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mon": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_osd": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_client_mode": "crc", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_mode": "crc", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_type": "", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_compress_secure": "false", 2024-04-22T06:15:40.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_idle_timeout": "900", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_ready_timeout": "10", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_data": "true", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_header": "true", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bad_msg": "false", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bug": "true", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_old_message": "true", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_skipped_message": "false", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_coremask": "0xF", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_enable_tso": "true", 2024-04-22T06:15:40.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hugepages": "", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_lro": "true", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_memory_channel": "4", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_pmd": "", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_port_id": "0", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_on_send": "false", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_initial_backoff": "0.200000", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_max": "1.000000", 2024-04-22T06:15:40.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_type": "", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_network_congestion": "0", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_socket_failures": "0", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_learn_addr_from_peer": "true", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_accept_failures": "4", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_backoff": "15.000000", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_client_mode": "crc", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_cluster_mode": "crc", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_service_mode": "crc", 2024-04-22T06:15:40.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_min_size": "1024", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_mode": "none", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_min_cost": "65536", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_public_type": "", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_service_mode": "crc", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_listen_backlog": "512", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_nodelay": "true", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_rcvbuf": "0", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_type": "async+posix", 2024-04-22T06:15:40.131 INFO:teuthology.orchestra.run.smithi028.stdout: "no_config_file": "false", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_completion_locks_per_session": "32", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_ops": "1024", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_tick_interval": "5.000000", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_timeout": "10.000000", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_blackhole": "false", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "openssl_engine_opts": "", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_delay_time": "5.000000", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_hist_halflife": "1000", 2024-04-22T06:15:40.132 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_low_ops": "2", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_ops": "4", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_slop": "0.020000", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_asio_thread_count": "2", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_async_recovery_min_cost": "100", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_max": "512", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_min": "64", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_degraded": "false", 2024-04-22T06:15:40.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_peering": "false", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_unfound": "true", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_beacon_report_interval": "300", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_duration": "30", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_max_block_size": "67108864", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blkin_trace_all": "false", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_for_log_corruption": "false", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-22T06:15:40.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_update_on_start": "true", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_cap": "256", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_size_cap": "524288000", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_op_priority": "63", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_watch_timeout": "30", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_command_max_records": "256", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_compact_on_start": "false", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_on_start": "true", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_weight_set": "true", 2024-04-22T06:15:40.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_feed_pullee": "-1", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_misdirected_ops": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_acting_change": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_purge_strays": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_op_order": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-22T06:15:40.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_shutdown": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_trim_objects": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_snaps": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_keys": "1024", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-22T06:15:40.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_stride": "524288", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_notify_timeout": "30", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep": "0.000000", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_discard_disconnected_ops": "true", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_enable_op_tracker": "true", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-22T06:15:40.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_info": "true", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown": "true", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_function_tracing": "false", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_grace": "20", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_interval": "6", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_peers": "10", 2024-04-22T06:15:40.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_size": "2000", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_stale": "600", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_max_size": "100000", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_min_size": "1000", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_size": "5120", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_kill_backfill_at": "0", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-22T06:15:40.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_cache_size": "50", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_dedup": "true", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max": "40", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_share_max_epochs": "40", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_name_len": "100", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_size": "0", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_backfills": "1", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_count": "5", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_period": "600", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_name_len": "2048", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_namespace_len": "256", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_size": "134217728", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-22T06:15:40.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_blocked_by": "16", 2024-04-22T06:15:40.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_log_entries": "10000", 2024-04-22T06:15:40.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-22T06:15:40.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pgls": "1024", 2024-04-22T06:15:40.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_cost": "8388608", 2024-04-22T06:15:40.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_objects": "10", 2024-04-22T06:15:40.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_scrubs": "3", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_trimming_pgs": "2", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_op_reply_len": "64", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_size": "90", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_profile": "balanced", 2024-04-22T06:15:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_base": "805306368", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_min": "134217728", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-22T06:15:40.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target": "4294967296", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_autotune": "false", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_pg_log_entries": "250", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_recovery_priority": "0", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_interval": "5", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_cache_shards": "32", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_op_tracker_shard": "32", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_auto_affinity": "true", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_node": "-1", 2024-04-22T06:15:40.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_prefer_iface": "true", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objecter_finishers": "1", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore": "bluestore", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_fuse": "false", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_tracing": "false", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_complaint_time": "30.000000", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_duration": "600", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_size": "20", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_size": "20", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_log_threshold": "5", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards": "0", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_hdd": "5", 2024-04-22T06:15:40.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_ssd": "8", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_min_cost": "65536", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue_cut_off": "high", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_timeout": "15", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_open_classes_on_start": "true", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_os_flags": "0", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_peering_op_priority": "255", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_cost": "1048576", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_priority": "5", 2024-04-22T06:15:40.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_max": "10000", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_min": "100", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-22T06:15:40.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crimson": "true", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flags": "0", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_min_size": "2", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_num": "32", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pgp_num": "0", 2024-04-22T06:15:40.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_ratio": "70", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_size": "3", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_type": "replicated", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_push_per_object_cost": "1000", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap": "true", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_cost": "20971520", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active": "0", 2024-04-22T06:15:40.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_single_start": "1", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_priority": "3", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_priority": "5", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep": "0.000000", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-22T06:15:40.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_repair_during_recovery": "false", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_requested_scrub_priority": "120", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair": "false", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_hour": "0", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_week_day": "0", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_max": "25", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_min": "5", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_cost": "52428800", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_during_recovery": "false", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_hour": "0", 2024-04-22T06:15:40.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_week_day": "0", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_event_cost": "4096", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_invalid_stats": "true", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_preemptions": "5", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_priority": "5", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_sleep": "0.000000", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-22T06:15:40.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_skip_data_digest": "false", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_smart_report_timeout": "5", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_cost": "1048576", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_priority": "5", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_transaction_size": "30", 2024-04-22T06:15:40.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tracing": "false", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_use_stale_snap": "false", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osdc_blkin_trace_all": "false", 2024-04-22T06:15:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_kill_at": "0", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_max_join_drift": "10", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min": "500", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min_wait": "0.050000", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_propose_interval": "1.000000", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max": "500", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_min": "10", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_stash_full_interval": "25", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_max": "500", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_min": "250", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "perf": "true", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-22T06:15:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "-", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrv": "", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "public_bind_addr": "-", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network": "", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network_interface": "", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_enabled": "false", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_session_max_number": "256", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_mon_op_timeout": "0", 2024-04-22T06:15:40.157 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_osd_op_timeout": "0", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_tracing": "false", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_atime_update_interval": "60", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_parent_reads": "false", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_snap_reads": "false", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blkin_trace_all": "false", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache": "true", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_policy": "writearound", 2024-04-22T06:15:40.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_size": "33554432", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_clone_copy_on_read": "false", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_compression_hint": "none", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_concurrent_management_ops": "10", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_clone_format": "auto", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_data_pool": "", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_features": "61", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_format": "2", 2024-04-22T06:15:40.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_map_options": "", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_order": "22", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_pool": "rbd", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_count": "0", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_unit": "0", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_enable_alloc_hint": "true", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler": "simple", 2024-04-22T06:15:40.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_order": "24", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_pool": "", 2024-04-22T06:15:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_splay_width": "4", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_parent_reads": "false", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_snap_reads": "false", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-22T06:15:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-22T06:15:40.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mtime_update_interval": "60", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_non_blocking_aio": "true", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_thread_timeout": "60", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_threads": "1", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_parent_cache_enabled": "false", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_plugins": "", 2024-04-22T06:15:40.164 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_limit": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_exclude_ops": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_limit": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-22T06:15:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_read_from_replica_policy": "default", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_skip_partial_discard": "true", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_tracing": "false", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_names": "true", 2024-04-22T06:15:40.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_pool": "true", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_base_url": "", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_log_level": "", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_acl_grants_max_num": "100", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_admin_entry": "admin", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_backend_store": "rados", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_barbican_url": "", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_beast_enable_async": "true", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_counters_cache": "false", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-22T06:15:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_enabled": "true", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_expiry_interval": "900", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_lru_size": "10000", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_config_store": "rados", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_content_length_compat": "false", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress": "true", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cors_rules_max_num": "100", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cross_domain_policy": "", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_addr": "", 2024-04-22T06:15:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_password": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_username": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_require_ssl": "true", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-22T06:15:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_addr": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_auth": "token", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_namespace": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_prefix": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-22T06:15:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_token_file": "", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_buffersize": "524288", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_time": "300", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-22T06:15:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_port": "6379", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_changes_size": "1000", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_num_shards": "128", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_window": "30", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-22T06:15:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-22T06:15:40.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_name": "", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_s3website_name": "", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dynamic_resharding": "true", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_gc_threads": "true", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_lc_threads": "true", 2024-04-22T06:15:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_ops_log": "false", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_quota_threads": "true", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_static_website": "false", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_usage_log": "false", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enforce_swift_acls": "true", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_exit_timeout_secs": "120", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_expose_bucket": "false", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_extended_http_attrs": "", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_filter": "none", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontends": "beast port=7480", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred": "50", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_objs": "32", 2024-04-22T06:15:40.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_period": "3600", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_init_timeout": "300", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_connection_idle": "300", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-22T06:15:40.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_domain": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password_path": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_project": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_tenant": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token_path": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_user": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_api_version": "2", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_domain": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_password": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_project": "", 2024-04-22T06:15:40.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_user": "", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_url": "", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_debug_interval": "-1", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_lock_max_time": "90", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_objs": "32", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_rules": "1000", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_worker": "3", 2024-04-22T06:15:40.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_thread_delay": "0", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_dnattr": "uid", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchfilter": "", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_uri": "", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_http_headers": "", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-22T06:15:40.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name_utc": "false", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_name_len": "0", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_size": "0", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_chunk_size": "4194304", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_listing_results": "1000", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_notify_retries": "10", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_param_size": "1048576", 2024-04-22T06:15:40.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_size": "5368709120", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_slo_entries": "1000", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_log_max_shards": "64", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mp_lock_max_time": "600", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-22T06:15:40.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_max_gc": "300", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_async_rados_threads": "32", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_control_oids": "8", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_numa_node": "-1", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-22T06:15:40.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_chunk_size": "100", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_gc_interval": "600", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_timeout": "600", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_tracing": "false", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_token": "", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_url": "", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_verify_ssl": "true", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_rados": "false", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_socket_path": "", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-22T06:15:40.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval": "2.000000", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_lanes": "3", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_partitions": "3", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_continue": "true", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-22T06:15:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_tracing": "false", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm": "", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_id": "", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region": "", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_request_uri": "", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_batch_size": "64", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-22T06:15:40.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_max_aio": "128", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_num_logs": "16", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_thread_interval": "600", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_resolve_cname": "false", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_run_sync_thread": "true", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-22T06:15:40.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_scheduler_type": "throttler", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_script_uri": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_service_provider_name": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_id": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_secret": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_entry": "sts", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_key": "sts", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_min_session_duration": "900", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_token_introspection_url": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_account_in_url": "false", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_entry": "auth", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_url": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_custom_header": "", 2024-04-22T06:15:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_need_stats": "true", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_tenant_name": "", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_token_expiration": "86400", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url": "", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url_prefix": "swift", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_lease_period": "120", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-22T06:15:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_thread_pool_size": "512", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_comment": "", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_createby": "", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_encoding": "", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_flag": "false", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_origin": "", 2024-04-22T06:15:40.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_tracker": "", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_trust_forwarded_https": "false", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_shards": "32", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_user_shards": "1", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_use_opa_authz": "false", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_counters_cache": "false", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_max_buckets": "1000", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_policies_max_num": "100", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-22T06:15:40.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_unique_email": "true", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_verify_ssl": "true", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone": "", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_id": "", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup": "", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_id": "", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_auto_session_clear": "true", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_block_size": "4096", 2024-04-22T06:15:40.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_size": "536870912", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-22T06:15:40.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_index_type": "binary_search", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_partition_filters": "false", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_perf": "false", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "run_dir": "/var/run/ceph", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_block_create": "true", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cache_lru_size": "67108864", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_max_object_size": "16777216", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-22T06:15:40.193 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_device_size": "53687091200", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_capacity": "16", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_main_device_type": "SSD", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_segment_size": "67108864", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "set_keepcaps": "false", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "setgroup": "", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser": "", 2024-04-22T06:15:40.194 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser_match_path": "", 2024-04-22T06:15:40.195 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-22T06:15:40.195 INFO:teuthology.orchestra.run.smithi028.stdout: "thp": "false", 2024-04-22T06:15:40.195 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_default_timeout": "60", 2024-04-22T06:15:40.195 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-22T06:15:40.195 INFO:teuthology.orchestra.run.smithi028.stdout: "throttler_perf_counter": "true" 2024-04-22T06:15:40.195 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-22T06:15:40.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-22T06:15:40.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:40.307+0000 7fd2d6a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:40.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:40.313+0000 7fd2d6a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:40.329 INFO:teuthology.orchestra.run.smithi028.stdout:{} 2024-04-22T06:15:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T06:15:40.340 DEBUG:teuthology.orchestra.run.smithi103:> type ceph-objectstore-tool 2024-04-22T06:15:40.356 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-22T06:15:40.356 DEBUG:teuthology.orchestra.run.smithi028:> type ceph-objectstore-tool 2024-04-22T06:15:40.371 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-22T06:15:40.371 INFO:teuthology.run_tasks:Running task radosbench... 2024-04-22T06:15:40.403 INFO:tasks.radosbench:Beginning radosbench... 2024-04-22T06:15:40.403 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-22T06:15:40.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-22T06:15:40.404 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-22T06:15:40.404 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-22T06:15:40.404 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:15:40.404 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3] 2024-04-22T06:15:40.404 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T06:15:40.404 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:15:40.405 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-22T06:15:40.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:15:40.455 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T06:15:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:40.522+0000 7f3c5fe06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:40.528+0000 7f3c5fe06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:40.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:40.522+0000 7f6895a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:40.534+0000 7f6895a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:41.347 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_0' created 2024-04-22T06:15:41.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-22T06:15:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:41.512+0000 7f098959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:41.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:41.526+0000 7f098959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-22T06:15:43.377 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 150 write' 2024-04-22T06:15:43.379 INFO:teuthology.run_tasks:Running task admin_socket... 2024-04-22T06:15:43.404 DEBUG:tasks.admin_socket:Running admin socket tests on client.0 2024-04-22T06:15:43.404 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -- /home/ubuntu/cephtest/admin_socket_client.0 && while test '!' -e /var/run/ceph/ceph-client.0.asok ; do sleep 1 ; done 2024-04-22T06:15:43.448 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-04-22T06:15:43.448 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 150 seconds or 0 objects 2024-04-22T06:15:43.448 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_29339 2024-04-22T06:15:43.448 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:15:43.448 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-04-22T06:15:44.423 DEBUG:tasks.admin_socket:Testing objecter_requests with config {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}', 'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base'} 2024-04-22T06:15:44.665 DEBUG:teuthology.orchestra.run.smithi028:> wget -q -O /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests -- 'http://git.ceph.com/?p=ceph-ci.git;a=blob_plain;f=src/test/admin_socket/objecter_requests;hb=ci-yingxin-crimson-fix-tri-mutex-4-base' && chmod u=rx -- /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-04-22T06:15:44.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 17 1 0.0624926 0.0625 0.02505 0.02505 2024-04-22T06:15:45.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-client.0.asok objecter_requests 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1, 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.6f779e7f", 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object0", 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.161 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object0", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449705+0000", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.162 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 11, 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.31f99cff", 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object10", 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object10", 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449978+0000", 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 14, 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.9a583caf", 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object13", 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object13", 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.166 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.450000+0000", 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 15, 2024-04-22T06:15:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.8197432a", 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object14", 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object14", 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.168 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.450007+0000", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 16, 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.46e9361a", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object15", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object15", 2024-04-22T06:15:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.450021+0000", 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.16a45765", 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object3", 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object3", 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.171 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449794+0000", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 6, 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.47a44fbd", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object5", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object5", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449847+0000", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.173 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2, 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.f04aaeb6", 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object1", 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object1", 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.174 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.175 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.175 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449758+0000", 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3, 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.bcd30843", 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object2", 2024-04-22T06:15:45.176 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object2", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449779+0000", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.177 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5, 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.ee662627", 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object4", 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object4", 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.178 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449824+0000", 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 8, 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.1c8a19c1", 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.179 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object7", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object7", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449866+0000", 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.180 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 9, 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.eee0b830", 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object8", 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object8", 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.572735s", 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.711951988, 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449877+0000", 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 10, 2024-04-22T06:15:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.afb74ba0", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object9", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object9", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.183 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449959+0000", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 12, 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.14fa8bf1", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object11", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object11", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449987+0000", 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 13, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.ec94f877", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object12", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object12", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.573735s", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.710952016, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.449993+0000", 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:15:45.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:15:45.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 17, 2024-04-22T06:15:45.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pg": "3.7c5e3e61", 2024-04-22T06:15:45.189 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-04-22T06:15:45.190 INFO:teuthology.orchestra.run.smithi028.stdout: "object_id": "benchmark_data_smithi028_29339_object16", 2024-04-22T06:15:45.190 INFO:teuthology.orchestra.run.smithi028.stdout: "object_locator": "@3", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_id": "benchmark_data_smithi028_29339_object16", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "target_object_locator": "@3", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "paused": 0, 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "used_replica": 0, 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "precalc_pgid": 0, 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "last_sent": "603.598734s", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.6859527169999999, 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "attempts": 1, 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "snapid": "head", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_context": "0=[]", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-04-22T06:15:43.474930+0000", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ops": [ 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-22T06:15:45.191 INFO:teuthology.orchestra.run.smithi028.stdout: "write 0~65536 in=65536b" 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ops": [], 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ops": [], 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_stat_ops": [], 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_ops": [], 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout: "command_ops": [] 2024-04-22T06:15:45.192 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-22T06:15:45.193 DEBUG:tasks.admin_socket:admin socket command objecter_requests returned { "ops": [ { "tid": 1, "pg": "3.6f779e7f", "osd": 0, "object_id": "benchmark_data_smithi028_29339_object0", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object0", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449705+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 11, "pg": "3.31f99cff", "osd": 0, "object_id": "benchmark_data_smithi028_29339_object10", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object10", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449978+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 14, "pg": "3.9a583caf", "osd": 0, "object_id": "benchmark_data_smithi028_29339_object13", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object13", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.450000+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 15, "pg": "3.8197432a", "osd": 0, "object_id": "benchmark_data_smithi028_29339_object14", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object14", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.450007+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 16, "pg": "3.46e9361a", "osd": 0, "object_id": "benchmark_data_smithi028_29339_object15", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object15", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.450021+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 4, "pg": "3.16a45765", "osd": 1, "object_id": "benchmark_data_smithi028_29339_object3", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object3", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449794+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 6, "pg": "3.47a44fbd", "osd": 1, "object_id": "benchmark_data_smithi028_29339_object5", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object5", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449847+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 2, "pg": "3.f04aaeb6", "osd": 2, "object_id": "benchmark_data_smithi028_29339_object1", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object1", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449758+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 3, "pg": "3.bcd30843", "osd": 2, "object_id": "benchmark_data_smithi028_29339_object2", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object2", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449779+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 5, "pg": "3.ee662627", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object4", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object4", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449824+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 8, "pg": "3.1c8a19c1", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object7", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object7", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449866+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 9, "pg": "3.eee0b830", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object8", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object8", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.572735s", "age": 1.711951988, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449877+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 10, "pg": "3.afb74ba0", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object9", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object9", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449959+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 12, "pg": "3.14fa8bf1", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object11", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object11", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449987+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 13, "pg": "3.ec94f877", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object12", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object12", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.573735s", "age": 1.710952016, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.449993+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 17, "pg": "3.7c5e3e61", "osd": 3, "object_id": "benchmark_data_smithi028_29339_object16", "object_locator": "@3", "target_object_id": "benchmark_data_smithi028_29339_object16", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "603.598734s", "age": 1.6859527169999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-22T06:15:43.474930+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } 2024-04-22T06:15:45.194 DEBUG:teuthology.orchestra.run.smithi028:> /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-04-22T06:15:45.237 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/admin_socket_client.0 2024-04-22T06:15:45.293 DEBUG:teuthology.parallel:result is None 2024-04-22T06:15:45.293 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2024-04-22T06:15:45.313 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) 2024-04-22T06:15:45.313 INFO:teuthology.orchestra.run:waiting for 4800 2024-04-22T06:15:45.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 17 1 0.0312455 0 - 0.02505 2024-04-22T06:15:45.456 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-22T06:15:45.456 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:15:45.456 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-22T06:15:45.456 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T06:15:45.456 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-22T06:15:45.459 INFO:tasks.ceph.osd.3:Started 2024-04-22T06:15:45.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T06:15:45.570 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:15:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:15:45.573 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T06:15:46.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 17 1 0.0208305 0 - 0.02505 2024-04-22T06:15:47.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 17 1 0.0156229 0 - 0.02505 2024-04-22T06:15:48.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 17 1 0.0124982 0 - 0.02505 2024-04-22T06:15:49.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 17 1 0.0104152 0 - 0.02505 2024-04-22T06:15:50.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 20 4 0.0357095 0.03125 0.0215071 3.02347 2024-04-22T06:15:50.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T06:15:50.675 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-22T06:15:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "ops_in_flight": [] 2024-04-22T06:15:50.675 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-22T06:15:51.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 20 4 0.0312459 0 - 3.02347 2024-04-22T06:15:52.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 40 24 0.166645 0.625 0.0146573 3.01167 2024-04-22T06:15:52.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:15:52.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:52.686+0000 7fda9855d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:52.693+0000 7fda9855d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:53.449 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 40 24 0.149981 0 - 3.01167 2024-04-22T06:15:54.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 210 194 1.10214 5.3125 0.136767 0.903003 2024-04-22T06:15:54.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:15:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:54.959+0000 7fdc4982d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:54.968+0000 7fdc4982d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:55.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 406 390 2.03101 12.25 0.0422233 0.490052 2024-04-22T06:15:55.685 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-22T06:15:55.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:15:55.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:15:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:55.846+0000 7f0c3c2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:55.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:55.858+0000 7f0c3c2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:15:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:15:56.167 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-04-22T06:15:54.254164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108036,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379324,"statfs":{"total":386547056640,"available":386436427776,"internally_reserved":0,"allocated":745472,"data_stored":2120790,"data_compressed":99054,"data_compressed_allocated":135168,"data_compressed_original":1916928,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-393760,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-134,"num_write_kb":-1702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.887194"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:49.449080+0000","last_change":"2024-04-22T06:15:49.449080+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:49.449080+0000","last_undegraded":"2024-04-22T06:15:49.449080+0000","last_fullsized":"2024-04-22T06:15:49.449080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"18'2","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-22T06:15:49.433359+0000","last_change":"2024-04-22T06:15:42.436819+0000","last_active":"2024-04-22T06:15:49.433359+0000","last_peered":"2024-04-22T06:15:49.433359+0000","last_clean":"2024-04-22T06:15:49.433359+0000","last_became_active":"2024-04-22T06:15:42.429697+0000","last_became_peered":"2024-04-22T06:15:42.429697+0000","last_unstale":"2024-04-22T06:15:49.433359+0000","last_undegraded":"2024-04-22T06:15:49.433359+0000","last_fullsized":"2024-04-22T06:15:49.433359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'1","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-22T06:15:49.405353+0000","last_change":"2024-04-22T06:15:42.433584+0000","last_active":"2024-04-22T06:15:49.405353+0000","last_peered":"2024-04-22T06:15:49.405353+0000","last_clean":"2024-04-22T06:15:49.405353+0000","last_became_active":"2024-04-22T06:15:42.423496+0000","last_became_peered":"2024-04-22T06:15:42.423496+0000","last_unstale":"2024-04-22T06:15:49.405353+0000","last_undegraded":"2024-04-22T06:15:49.405353+0000","last_fullsized":"2024-04-22T06:15:49.405353+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":3,"reported_epoch":18,"state":"peering","last_fresh":"2024-04-22T06:15:28.275179+0000","last_change":"2024-04-22T06:15:49.391374+0000","last_active":"2024-04-22T06:15:27.257222+0000","last_peered":"2024-04-22T06:15:27.257222+0000","last_clean":"2024-04-22T06:15:27.257222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:28.275179+0000","last_undegraded":"2024-04-22T06:15:28.275179+0000","last_fullsized":"2024-04-22T06:15:28.275179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-22T06:15:49.410763+0000","last_change":"2024-04-22T06:15:30.446240+0000","last_active":"2024-04-22T06:15:49.410763+0000","last_peered":"2024-04-22T06:15:49.410763+0000","last_clean":"2024-04-22T06:15:49.410763+0000","last_became_active":"2024-04-22T06:15:30.439253+0000","last_became_peered":"2024-04-22T06:15:30.439253+0000","last_unstale":"2024-04-22T06:15:49.410763+0000","last_undegraded":"2024-04-22T06:15:49.410763+0000","last_fullsized":"2024-04-22T06:15:49.410763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-22T06:15:49.416939+0000","last_change":"2024-04-22T06:15:30.429530+0000","last_active":"2024-04-22T06:15:49.416939+0000","last_peered":"2024-04-22T06:15:49.416939+0000","last_clean":"2024-04-22T06:15:49.416939+0000","last_became_active":"2024-04-22T06:15:30.424410+0000","last_became_peered":"2024-04-22T06:15:30.424410+0000","last_unstale":"2024-04-22T06:15:49.416939+0000","last_undegraded":"2024-04-22T06:15:49.416939+0000","last_fullsized":"2024-04-22T06:15:49.416939+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:49.446785+0000","last_change":"2024-04-22T06:15:49.446785+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:49.446785+0000","last_undegraded":"2024-04-22T06:15:49.446785+0000","last_fullsized":"2024-04-22T06:15:49.446785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"peering","last_fresh":"2024-04-22T06:15:43.376133+0000","last_change":"2024-04-22T06:15:49.404560+0000","last_active":"2024-04-22T06:15:49.404560+0000","last_peered":"2024-04-22T06:15:43.376133+0000","last_clean":"2024-04-22T06:15:43.376133+0000","last_became_active":"2024-04-22T06:15:30.397935+0000","last_became_peered":"2024-04-22T06:15:30.397935+0000","last_unstale":"2024-04-22T06:15:43.376133+0000","last_undegraded":"2024-04-22T06:15:43.376133+0000","last_fullsized":"2024-04-22T06:15:43.376133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:49.457509+0000","last_change":"2024-04-22T06:15:49.457509+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:49.457509+0000","last_undegraded":"2024-04-22T06:15:49.457509+0000","last_fullsized":"2024-04-22T06:15:49.457509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-22T06:15:49.404218+0000","last_change":"2024-04-22T06:15:30.424826+0000","last_active":"2024-04-22T06:15:49.404218+0000","last_peered":"2024-04-22T06:15:49.404218+0000","last_clean":"2024-04-22T06:15:49.404218+0000","last_became_active":"2024-04-22T06:15:30.416975+0000","last_became_peered":"2024-04-22T06:15:30.416975+0000","last_unstale":"2024-04-22T06:15:49.404218+0000","last_undegraded":"2024-04-22T06:15:49.404218+0000","last_fullsized":"2024-04-22T06:15:49.404218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"peering","last_fresh":"2024-04-22T06:15:43.383490+0000","last_change":"2024-04-22T06:15:49.423955+0000","last_active":"2024-04-22T06:15:49.423955+0000","last_peered":"2024-04-22T06:15:43.383490+0000","last_clean":"2024-04-22T06:15:43.383490+0000","last_became_active":"2024-04-22T06:15:30.406142+0000","last_became_peered":"2024-04-22T06:15:30.406142+0000","last_unstale":"2024-04-22T06:15:43.383490+0000","last_undegraded":"2024-04-22T06:15:43.383490+0000","last_fullsized":"2024-04-22T06:15:43.383490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3,"reported_epoch":18,"state":"peering","last_fresh":"2024-04-22T06:15:30.332484+0000","last_change":"2024-04-22T06:15:49.414734+0000","last_active":"2024-04-22T06:15:29.310548+0000","last_peered":"2024-04-22T06:15:29.310548+0000","last_clean":"2024-04-22T06:15:29.310548+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:30.332484+0000","last_undegraded":"2024-04-22T06:15:30.332484+0000","last_fullsized":"2024-04-22T06:15:30.332484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:49.494474+0000","last_change":"2024-04-22T06:15:49.494474+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:49.494474+0000","last_undegraded":"2024-04-22T06:15:49.494474+0000","last_fullsized":"2024-04-22T06:15:49.494474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"peering","last_fresh":"2024-04-22T06:15:43.383449+0000","last_change":"2024-04-22T06:15:49.423672+0000","last_active":"2024-04-22T06:15:49.423672+0000","last_peered":"2024-04-22T06:15:43.383449+0000","last_clean":"2024-04-22T06:15:43.383449+0000","last_became_active":"2024-04-22T06:15:30.412641+0000","last_became_peered":"2024-04-22T06:15:30.412641+0000","last_unstale":"2024-04-22T06:15:43.383449+0000","last_undegraded":"2024-04-22T06:15:43.383449+0000","last_fullsized":"2024-04-22T06:15:43.383449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-04-22T06:15:41.344834+0000","last_change":"2024-04-22T06:15:41.344834+0000","last_active":"2024-04-22T06:15:41.344834+0000","last_peered":"2024-04-22T06:15:41.344834+0000","last_clean":"2024-04-22T06:15:41.344834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T06:15:41.344834+0000","last_undegraded":"2024-04-22T06:15:41.344834+0000","last_fullsized":"2024-04-22T06:15:41.344834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-22T06:15:49.412402+0000","last_change":"2024-04-22T06:15:30.460326+0000","last_active":"2024-04-22T06:15:49.412402+0000","last_peered":"2024-04-22T06:15:49.412402+0000","last_clean":"2024-04-22T06:15:49.412402+0000","last_became_active":"2024-04-22T06:15:30.452351+0000","last_became_peered":"2024-04-22T06:15:30.452351+0000","last_unstale":"2024-04-22T06:15:49.412402+0000","last_undegraded":"2024-04-22T06:15:49.412402+0000","last_fullsized":"2024-04-22T06:15:49.412402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":786432,"data_compressed":37317,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":34,"acting":34,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":20,"acting":20,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59818,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27028,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344812,"statfs":{"total":96636764160,"available":96609087488,"internally_reserved":0,"allocated":208896,"data_stored":696777,"data_compressed":33018,"data_compressed_allocated":45056,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26968,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344872,"statfs":{"total":96636764160,"available":96609148928,"internally_reserved":0,"allocated":147456,"data_stored":106409,"data_compressed":3109,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27028,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344812,"statfs":{"total":96636764160,"available":96609087488,"internally_reserved":0,"allocated":208896,"data_stored":696777,"data_compressed":33018,"data_compressed_allocated":45056,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":262144,"data_compressed":12439,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":262144,"data_compressed":12439,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":262144,"data_compressed":12439,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:15:56.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 605 589 2.8314 12.4375 0.0463144 0.352173 2024-04-22T06:15:57.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:15:57.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 804 788 3.51745 12.4375 0.0675575 0.283489 2024-04-22T06:15:57.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:57.464+0000 7fc647cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:57.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:57.476+0000 7fc647cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:58.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 15 978 963 4.01205 10.9375 0.116747 0.248489 2024-04-22T06:15:59.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 1165 1149 4.48778 11.625 0.141343 0.222225 2024-04-22T06:15:59.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:15:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:59.867+0000 7fda57ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:15:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:15:59.876+0000 7fda57ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:00.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 1359 1343 4.93695 12.125 0.0745016 0.202036 2024-04-22T06:16:01.169 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-22T06:16:01.169 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:01.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:16:01.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:01.319+0000 7f962919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:01.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:01.333+0000 7f962919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:01.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 1559 1543 5.35705 12.5 0.0586307 0.186318 2024-04-22T06:16:01.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:16:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:16:01.655 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-04-22T06:16:00.255220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38339104,"num_objects":578,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":713,"num_write_kb":38758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":657,"ondisk_log_size":657,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":117940,"kb_used_data":6416,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":377369420,"statfs":{"total":386547056640,"available":386426286080,"internally_reserved":0,"allocated":6569984,"data_stored":92888228,"data_compressed":4404477,"data_compressed_allocated":5804032,"data_compressed_original":92618752,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38076960,"num_objects":574,"num_object_clones":0,"num_object_copies":1722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":709,"num_write_kb":38502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.883326"},"pg_stats":[{"pgid":"3.f","version":"21'28","reported_seq":45,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.203283+0000","last_change":"2024-04-22T06:15:51.535058+0000","last_active":"2024-04-22T06:15:55.203283+0000","last_peered":"2024-04-22T06:15:55.203283+0000","last_clean":"2024-04-22T06:15:55.203283+0000","last_became_active":"2024-04-22T06:15:51.528948+0000","last_became_peered":"2024-04-22T06:15:51.528948+0000","last_unstale":"2024-04-22T06:15:55.203283+0000","last_undegraded":"2024-04-22T06:15:55.203283+0000","last_fullsized":"2024-04-22T06:15:55.203283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'19","reported_seq":36,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.165513+0000","last_change":"2024-04-22T06:15:51.521546+0000","last_active":"2024-04-22T06:15:55.165513+0000","last_peered":"2024-04-22T06:15:55.165513+0000","last_clean":"2024-04-22T06:15:55.165513+0000","last_became_active":"2024-04-22T06:15:51.513951+0000","last_became_peered":"2024-04-22T06:15:51.513951+0000","last_unstale":"2024-04-22T06:15:55.165513+0000","last_undegraded":"2024-04-22T06:15:55.165513+0000","last_fullsized":"2024-04-22T06:15:55.165513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'16","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.087675+0000","last_change":"2024-04-22T06:15:51.536930+0000","last_active":"2024-04-22T06:15:55.087675+0000","last_peered":"2024-04-22T06:15:55.087675+0000","last_clean":"2024-04-22T06:15:55.087675+0000","last_became_active":"2024-04-22T06:15:51.528251+0000","last_became_peered":"2024-04-22T06:15:51.528251+0000","last_unstale":"2024-04-22T06:15:55.087675+0000","last_undegraded":"2024-04-22T06:15:55.087675+0000","last_fullsized":"2024-04-22T06:15:55.087675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'25","reported_seq":42,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.150391+0000","last_change":"2024-04-22T06:15:51.506665+0000","last_active":"2024-04-22T06:15:55.150391+0000","last_peered":"2024-04-22T06:15:55.150391+0000","last_clean":"2024-04-22T06:15:55.150391+0000","last_became_active":"2024-04-22T06:15:51.501835+0000","last_became_peered":"2024-04-22T06:15:51.501835+0000","last_unstale":"2024-04-22T06:15:55.150391+0000","last_undegraded":"2024-04-22T06:15:55.150391+0000","last_fullsized":"2024-04-22T06:15:55.150391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'69","reported_seq":82,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:59.281003+0000","last_change":"2024-04-22T06:15:51.614224+0000","last_active":"2024-04-22T06:15:59.281003+0000","last_peered":"2024-04-22T06:15:59.281003+0000","last_clean":"2024-04-22T06:15:59.281003+0000","last_became_active":"2024-04-22T06:15:51.603643+0000","last_became_peered":"2024-04-22T06:15:51.603643+0000","last_unstale":"2024-04-22T06:15:59.281003+0000","last_undegraded":"2024-04-22T06:15:59.281003+0000","last_fullsized":"2024-04-22T06:15:59.281003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'30","reported_seq":55,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.082137+0000","last_change":"2024-04-22T06:15:42.436819+0000","last_active":"2024-04-22T06:15:55.082137+0000","last_peered":"2024-04-22T06:15:55.082137+0000","last_clean":"2024-04-22T06:15:55.082137+0000","last_became_active":"2024-04-22T06:15:42.429697+0000","last_became_peered":"2024-04-22T06:15:42.429697+0000","last_unstale":"2024-04-22T06:15:55.082137+0000","last_undegraded":"2024-04-22T06:15:55.082137+0000","last_fullsized":"2024-04-22T06:15:55.082137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'21","reported_seq":46,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.137814+0000","last_change":"2024-04-22T06:15:42.433584+0000","last_active":"2024-04-22T06:15:55.137814+0000","last_peered":"2024-04-22T06:15:55.137814+0000","last_clean":"2024-04-22T06:15:55.137814+0000","last_became_active":"2024-04-22T06:15:42.423496+0000","last_became_peered":"2024-04-22T06:15:42.423496+0000","last_unstale":"2024-04-22T06:15:55.137814+0000","last_undegraded":"2024-04-22T06:15:55.137814+0000","last_fullsized":"2024-04-22T06:15:55.137814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'28","reported_seq":45,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.194728+0000","last_change":"2024-04-22T06:15:51.582508+0000","last_active":"2024-04-22T06:15:55.194728+0000","last_peered":"2024-04-22T06:15:55.194728+0000","last_clean":"2024-04-22T06:15:55.194728+0000","last_became_active":"2024-04-22T06:15:51.572917+0000","last_became_peered":"2024-04-22T06:15:51.572917+0000","last_unstale":"2024-04-22T06:15:55.194728+0000","last_undegraded":"2024-04-22T06:15:55.194728+0000","last_fullsized":"2024-04-22T06:15:55.194728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":151,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.489232+0000","last_change":"2024-04-22T06:15:51.606227+0000","last_active":"2024-04-22T06:15:53.489232+0000","last_peered":"2024-04-22T06:15:53.489232+0000","last_clean":"2024-04-22T06:15:53.489232+0000","last_became_active":"2024-04-22T06:15:51.599955+0000","last_became_peered":"2024-04-22T06:15:51.599955+0000","last_unstale":"2024-04-22T06:15:53.489232+0000","last_undegraded":"2024-04-22T06:15:53.489232+0000","last_fullsized":"2024-04-22T06:15:53.489232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.650438+0000","last_change":"2024-04-22T06:15:30.446240+0000","last_active":"2024-04-22T06:15:53.650438+0000","last_peered":"2024-04-22T06:15:53.650438+0000","last_clean":"2024-04-22T06:15:53.650438+0000","last_became_active":"2024-04-22T06:15:30.439253+0000","last_became_peered":"2024-04-22T06:15:30.439253+0000","last_unstale":"2024-04-22T06:15:53.650438+0000","last_undegraded":"2024-04-22T06:15:53.650438+0000","last_fullsized":"2024-04-22T06:15:53.650438+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'23","reported_seq":40,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.222349+0000","last_change":"2024-04-22T06:15:51.578726+0000","last_active":"2024-04-22T06:15:55.222349+0000","last_peered":"2024-04-22T06:15:55.222349+0000","last_clean":"2024-04-22T06:15:55.222349+0000","last_became_active":"2024-04-22T06:15:51.565082+0000","last_became_peered":"2024-04-22T06:15:51.565082+0000","last_unstale":"2024-04-22T06:15:55.222349+0000","last_undegraded":"2024-04-22T06:15:55.222349+0000","last_fullsized":"2024-04-22T06:15:55.222349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.633093+0000","last_change":"2024-04-22T06:15:30.429530+0000","last_active":"2024-04-22T06:15:53.633093+0000","last_peered":"2024-04-22T06:15:53.633093+0000","last_clean":"2024-04-22T06:15:53.633093+0000","last_became_active":"2024-04-22T06:15:30.424410+0000","last_became_peered":"2024-04-22T06:15:30.424410+0000","last_unstale":"2024-04-22T06:15:53.633093+0000","last_undegraded":"2024-04-22T06:15:53.633093+0000","last_fullsized":"2024-04-22T06:15:53.633093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'86","reported_seq":99,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:59.214942+0000","last_change":"2024-04-22T06:15:51.551585+0000","last_active":"2024-04-22T06:15:59.214942+0000","last_peered":"2024-04-22T06:15:59.214942+0000","last_clean":"2024-04-22T06:15:59.214942+0000","last_became_active":"2024-04-22T06:15:51.543197+0000","last_became_peered":"2024-04-22T06:15:51.543197+0000","last_unstale":"2024-04-22T06:15:59.214942+0000","last_undegraded":"2024-04-22T06:15:59.214942+0000","last_fullsized":"2024-04-22T06:15:59.214942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.639567+0000","last_change":"2024-04-22T06:15:51.511180+0000","last_active":"2024-04-22T06:15:53.639567+0000","last_peered":"2024-04-22T06:15:53.639567+0000","last_clean":"2024-04-22T06:15:53.639567+0000","last_became_active":"2024-04-22T06:15:51.501751+0000","last_became_peered":"2024-04-22T06:15:51.501751+0000","last_unstale":"2024-04-22T06:15:53.639567+0000","last_undegraded":"2024-04-22T06:15:53.639567+0000","last_fullsized":"2024-04-22T06:15:53.639567+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'61","reported_seq":74,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:59.215215+0000","last_change":"2024-04-22T06:15:51.578335+0000","last_active":"2024-04-22T06:15:59.215215+0000","last_peered":"2024-04-22T06:15:59.215215+0000","last_clean":"2024-04-22T06:15:59.215215+0000","last_became_active":"2024-04-22T06:15:51.572091+0000","last_became_peered":"2024-04-22T06:15:51.572091+0000","last_unstale":"2024-04-22T06:15:59.215215+0000","last_undegraded":"2024-04-22T06:15:59.215215+0000","last_fullsized":"2024-04-22T06:15:59.215215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4063232,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.642111+0000","last_change":"2024-04-22T06:15:30.424826+0000","last_active":"2024-04-22T06:15:53.642111+0000","last_peered":"2024-04-22T06:15:53.642111+0000","last_clean":"2024-04-22T06:15:53.642111+0000","last_became_active":"2024-04-22T06:15:30.416975+0000","last_became_peered":"2024-04-22T06:15:30.416975+0000","last_unstale":"2024-04-22T06:15:53.642111+0000","last_undegraded":"2024-04-22T06:15:53.642111+0000","last_fullsized":"2024-04-22T06:15:53.642111+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'12","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:54.900079+0000","last_change":"2024-04-22T06:15:51.496872+0000","last_active":"2024-04-22T06:15:54.900079+0000","last_peered":"2024-04-22T06:15:54.900079+0000","last_clean":"2024-04-22T06:15:54.900079+0000","last_became_active":"2024-04-22T06:15:51.489010+0000","last_became_peered":"2024-04-22T06:15:51.489010+0000","last_unstale":"2024-04-22T06:15:54.900079+0000","last_undegraded":"2024-04-22T06:15:54.900079+0000","last_fullsized":"2024-04-22T06:15:54.900079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'20","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:55.251794+0000","last_change":"2024-04-22T06:15:51.493998+0000","last_active":"2024-04-22T06:15:55.251794+0000","last_peered":"2024-04-22T06:15:55.251794+0000","last_clean":"2024-04-22T06:15:55.251794+0000","last_became_active":"2024-04-22T06:15:51.485462+0000","last_became_peered":"2024-04-22T06:15:51.485462+0000","last_unstale":"2024-04-22T06:15:55.251794+0000","last_undegraded":"2024-04-22T06:15:55.251794+0000","last_fullsized":"2024-04-22T06:15:55.251794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:52.401164+0000","last_change":"2024-04-22T06:15:51.565742+0000","last_active":"2024-04-22T06:15:52.401164+0000","last_peered":"2024-04-22T06:15:52.401164+0000","last_clean":"2024-04-22T06:15:52.401164+0000","last_became_active":"2024-04-22T06:15:51.559501+0000","last_became_peered":"2024-04-22T06:15:51.559501+0000","last_unstale":"2024-04-22T06:15:52.401164+0000","last_undegraded":"2024-04-22T06:15:52.401164+0000","last_fullsized":"2024-04-22T06:15:52.401164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'21","reported_seq":38,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:54.917907+0000","last_change":"2024-04-22T06:15:51.579389+0000","last_active":"2024-04-22T06:15:54.917907+0000","last_peered":"2024-04-22T06:15:54.917907+0000","last_clean":"2024-04-22T06:15:54.917907+0000","last_became_active":"2024-04-22T06:15:51.572492+0000","last_became_peered":"2024-04-22T06:15:51.572492+0000","last_unstale":"2024-04-22T06:15:54.917907+0000","last_undegraded":"2024-04-22T06:15:54.917907+0000","last_fullsized":"2024-04-22T06:15:54.917907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.487945+0000","last_change":"2024-04-22T06:15:51.565914+0000","last_active":"2024-04-22T06:15:53.487945+0000","last_peered":"2024-04-22T06:15:53.487945+0000","last_clean":"2024-04-22T06:15:53.487945+0000","last_became_active":"2024-04-22T06:15:51.555975+0000","last_became_peered":"2024-04-22T06:15:51.555975+0000","last_unstale":"2024-04-22T06:15:53.487945+0000","last_undegraded":"2024-04-22T06:15:53.487945+0000","last_fullsized":"2024-04-22T06:15:53.487945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'87","reported_seq":100,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:59.336508+0000","last_change":"2024-04-22T06:15:51.579037+0000","last_active":"2024-04-22T06:15:59.336508+0000","last_peered":"2024-04-22T06:15:59.336508+0000","last_clean":"2024-04-22T06:15:59.336508+0000","last_became_active":"2024-04-22T06:15:51.568061+0000","last_became_peered":"2024-04-22T06:15:51.568061+0000","last_unstale":"2024-04-22T06:15:59.336508+0000","last_undegraded":"2024-04-22T06:15:59.336508+0000","last_fullsized":"2024-04-22T06:15:59.336508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:52.393574+0000","last_change":"2024-04-22T06:15:51.541919+0000","last_active":"2024-04-22T06:15:52.393574+0000","last_peered":"2024-04-22T06:15:52.393574+0000","last_clean":"2024-04-22T06:15:52.393574+0000","last_became_active":"2024-04-22T06:15:51.537273+0000","last_became_peered":"2024-04-22T06:15:51.537273+0000","last_unstale":"2024-04-22T06:15:52.393574+0000","last_undegraded":"2024-04-22T06:15:52.393574+0000","last_fullsized":"2024-04-22T06:15:52.393574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'14","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:54.892724+0000","last_change":"2024-04-22T06:15:51.558610+0000","last_active":"2024-04-22T06:15:54.892724+0000","last_peered":"2024-04-22T06:15:54.892724+0000","last_clean":"2024-04-22T06:15:54.892724+0000","last_became_active":"2024-04-22T06:15:51.551084+0000","last_became_peered":"2024-04-22T06:15:51.551084+0000","last_unstale":"2024-04-22T06:15:54.892724+0000","last_undegraded":"2024-04-22T06:15:54.892724+0000","last_fullsized":"2024-04-22T06:15:54.892724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-22T06:15:53.649286+0000","last_change":"2024-04-22T06:15:30.460326+0000","last_active":"2024-04-22T06:15:53.649286+0000","last_peered":"2024-04-22T06:15:53.649286+0000","last_clean":"2024-04-22T06:15:53.649286+0000","last_became_active":"2024-04-22T06:15:30.452351+0000","last_became_peered":"2024-04-22T06:15:30.452351+0000","last_unstale":"2024-04-22T06:15:53.649286+0000","last_undegraded":"2024-04-22T06:15:53.649286+0000","last_fullsized":"2024-04-22T06:15:53.649286+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":37748736,"num_objects":576,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7417856,"data_stored":118685696,"data_compressed":5633823,"data_compressed_allocated":7417856,"data_compressed_original":118685696,"omap_allocated":0,"internal_metadata":0},"log_size":560,"ondisk_log_size":560,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59818,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35308,"kb_used_data":4260,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94336532,"statfs":{"total":96636764160,"available":96600608768,"internally_reserved":0,"allocated":4362240,"data_stored":66639793,"data_compressed":3162711,"data_compressed_allocated":4165632,"data_compressed_original":66568192,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27532,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":720896,"data_stored":8378289,"data_compressed":396940,"data_compressed_allocated":524288,"data_compressed_original":8306688,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27532,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":724992,"data_stored":8836497,"data_compressed":416787,"data_compressed_allocated":548864,"data_compressed_original":8781824,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27568,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":761856,"data_stored":9033649,"data_compressed":428039,"data_compressed_allocated":565248,"data_compressed_original":8962048,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1069056,"data_stored":17104896,"data_compressed":811893,"data_compressed_allocated":1069056,"data_compressed_original":17104896,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":18481152,"data_compressed":877215,"data_compressed_allocated":1155072,"data_compressed_original":18481152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1032192,"data_stored":16515072,"data_compressed":783905,"data_compressed_allocated":1032192,"data_compressed_original":16515072,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4161536,"data_stored":66584576,"data_compressed":3160810,"data_compressed_allocated":4161536,"data_compressed_original":66584576,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:16:02.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:16:02.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:02.397+0000 7f3ca595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:02.406+0000 7f3ca595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:02.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 1748 1732 5.69675 11.8125 0.0998236 0.174875 2024-04-22T06:16:03.450 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:16:03.451856+0000 min lat: 0.0146573 max lat: 10.177 avg lat: 0.164817 2024-04-22T06:16:03.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:16:03.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 1954 1938 6.05559 12.875 0.0885797 0.164817 2024-04-22T06:16:04.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 2151 2135 6.35349 12.3125 0.0540316 0.156986 2024-04-22T06:16:04.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:16:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:05.332+0000 7fe46eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:05.343+0000 7fe46eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:05.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 2329 2313 6.57032 11.125 0.0419342 0.151904 2024-04-22T06:16:06.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 2530 2514 6.8308 12.5625 0.0595032 0.146023 2024-04-22T06:16:06.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-22T06:16:06.657 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:06.658 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2024-04-22T06:16:06.658 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T06:16:06.658 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:16:06.725 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T06:16:07.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 2581 2565 6.67897 3.1875 0.056944 0.144826 2024-04-22T06:16:07.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:16:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:07.839+0000 7f316f95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:07.846+0000 7f316f95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:08.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 2581 2565 6.41181 0 - 0.144826 2024-04-22T06:16:09.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 2581 2565 6.16519 0 - 0.144826 2024-04-22T06:16:10.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 2581 2565 5.93684 0 - 0.144826 2024-04-22T06:16:11.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 2581 2565 5.7248 0 - 0.144826 2024-04-22T06:16:11.725 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-04-22T06:16:11.726 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:11.726 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-22T06:16:11.726 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T06:16:11.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T06:16:11.729 INFO:tasks.ceph.osd.1:Started 2024-04-22T06:16:11.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T06:16:11.843 INFO:teuthology.orchestra.run.smithi028.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:16:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:16:11.847 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T06:16:12.451 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 2581 2565 5.52739 0 - 0.144826 2024-04-22T06:16:13.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 2581 2565 5.34315 0 - 0.144826 2024-04-22T06:16:14.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 2581 2565 5.17078 0 - 0.144826 2024-04-22T06:16:15.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 2581 2565 5.00919 0 - 0.144826 2024-04-22T06:16:16.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 2581 2565 4.8574 0 - 0.144826 2024-04-22T06:16:16.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T06:16:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:16:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:16:16.951 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T06:16:17.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 2581 2565 4.71453 0 - 0.144826 2024-04-22T06:16:18.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 2581 2565 4.57983 0 - 0.144826 2024-04-22T06:16:19.452 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 2581 2565 4.45262 0 - 0.144826 2024-04-22T06:16:20.408 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T06:16:20.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 2581 2565 4.33228 0 - 0.144826 2024-04-22T06:16:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T06:16:20.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:16:21.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 2581 2565 4.21827 0 - 0.144826 2024-04-22T06:16:21.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T06:16:22.064 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "ops_in_flight": [ 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 67, 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.558175+0000" 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.558185+0000" 2024-04-22T06:16:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.558527+0000" 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.559184+0000" 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.560057+0000" 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 79, 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.068 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.576659+0000" 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.576688+0000" 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.577201+0000" 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.069 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.577486+0000" 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.578261+0000" 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 83, 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.581500+0000" 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.072 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.581512+0000" 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.581901+0000" 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.582395+0000" 2024-04-22T06:16:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.583614+0000" 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 70, 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.562868+0000" 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.562876+0000" 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.075 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.563056+0000" 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.563337+0000" 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.564680+0000" 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 62, 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.545627+0000" 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.545653+0000" 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.546732+0000" 2024-04-22T06:16:22.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.083 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.083 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.083 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.547328+0000" 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.549391+0000" 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 64, 2024-04-22T06:16:22.085 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.555546+0000" 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.555556+0000" 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.555784+0000" 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.556157+0000" 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.556525+0000" 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.089 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 74, 2024-04-22T06:16:22.089 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.565688+0000" 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.565697+0000" 2024-04-22T06:16:22.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.566396+0000" 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.566626+0000" 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.568026+0000" 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client_request", 2024-04-22T06:16:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 76, 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.574901+0000" 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.574913+0000" 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.094 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.575154+0000" 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.575309+0000" 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-04-22T06:16:21.575725+0000", 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "blocker": { 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "detail": { 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "waiting": 1, 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: "held_by_operation_id": 74 2024-04-22T06:16:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: "blocking_event": { 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time_event": { 2024-04-22T06:16:22.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:22.098 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:22.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-22T06:16:22.098 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-22T06:16:22.098 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-22T06:16:22.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 2581 2565 4.11012 0 - 0.144826 2024-04-22T06:16:22.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:16:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:22.673+0000 7f7fcfb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:22.682+0000 7f7fcfb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:23.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:16:23.454164+0000 min lat: 0.0146573 max lat: 16.2115 avg lat: 0.23606 2024-04-22T06:16:23.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:16:23.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 2725 2709 4.23234 0.5625 0.100806 0.23606 2024-04-22T06:16:24.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 2923 2907 4.43091 12.375 0.0611739 0.225416 2024-04-22T06:16:24.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:16:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:25.059+0000 7fe6e8b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:25.068+0000 7fe6e8b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:25.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 3109 3093 4.60216 11.625 0.106348 0.21707 2024-04-22T06:16:26.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 3293 3277 4.76255 11.5 0.152727 0.209707 2024-04-22T06:16:27.100 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-04-22T06:16:27.100 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:27.101 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 3, 1] 2024-04-22T06:16:27.101 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T06:16:27.101 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:16:27.172 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T06:16:27.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:16:27.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 3425 3409 4.84179 8.25 0.0222022 0.204865 2024-04-22T06:16:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:27.495+0000 7ff62735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:27.501+0000 7ff62735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:28.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 3425 3409 4.73419 0 - 0.204865 2024-04-22T06:16:29.453 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 3425 3409 4.63128 0 - 0.204865 2024-04-22T06:16:30.454 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 3425 3409 4.53274 0 - 0.204865 2024-04-22T06:16:31.454 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 3425 3409 4.4383 0 - 0.204865 2024-04-22T06:16:32.173 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 3, 1] 2024-04-22T06:16:32.174 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:32.174 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 3, 1] 2024-04-22T06:16:32.174 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T06:16:32.174 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:16:32.251 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T06:16:32.454 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 3425 3409 4.34772 0 - 0.204865 2024-04-22T06:16:33.454 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 3425 3409 4.26076 0 - 0.204865 2024-04-22T06:16:34.454 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 3425 3409 4.17721 0 - 0.204865 2024-04-22T06:16:35.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 3425 3409 4.09688 0 - 0.204865 2024-04-22T06:16:36.454 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 3425 3409 4.01957 0 - 0.204865 2024-04-22T06:16:37.252 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 1] live_osds: [0, 3] 2024-04-22T06:16:37.252 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:37.252 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-22T06:16:37.252 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T06:16:37.252 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T06:16:37.255 INFO:tasks.ceph.osd.2:Started 2024-04-22T06:16:37.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T06:16:37.365 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:16:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:16:37.369 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-22T06:16:37.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 3425 3409 3.94514 0 - 0.204865 2024-04-22T06:16:38.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 3425 3409 3.87341 0 - 0.204865 2024-04-22T06:16:39.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 3425 3409 3.80424 0 - 0.204865 2024-04-22T06:16:40.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 3425 3409 3.7375 0 - 0.204865 2024-04-22T06:16:41.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 3425 3409 3.67305 0 - 0.204865 2024-04-22T06:16:42.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T06:16:42.455 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 3425 3409 3.6108 0 - 0.204865 2024-04-22T06:16:42.471 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:16:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:16:42.474 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-22T06:16:43.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:16:43.456892+0000 min lat: 0.0145691 max lat: 16.2115 avg lat: 0.204865 2024-04-22T06:16:43.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:16:43.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 3425 3409 3.55062 0 - 0.204865 2024-04-22T06:16:44.041 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T06:16:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T06:16:44.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:16:44.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 3425 3409 3.49241 0 - 0.204865 2024-04-22T06:16:45.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 3425 3409 3.43608 0 - 0.204865 2024-04-22T06:16:46.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:16:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:46.292+0000 7f5a88ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:46.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:16:46.298+0000 7f5a88ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:46.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 3427 3411 3.38352 0.00657895 19.0822 0.215942 2024-04-22T06:16:47.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 3427 3411 3.33065 0 - 0.215942 2024-04-22T06:16:47.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T06:16:47.586 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "ops_in_flight": [ 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 64, 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.172214+0000" 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.587 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.172224+0000" 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.172386+0000" 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.172984+0000" 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.588 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.173202+0000" 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 67, 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.176698+0000" 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.176716+0000" 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.590 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.176955+0000" 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.178150+0000" 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.178424+0000" 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.591 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 75, 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.195013+0000" 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.195027+0000" 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.195417+0000" 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.196773+0000" 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.197980+0000" 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.594 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 70, 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.181305+0000" 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.181314+0000" 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.182154+0000" 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.182332+0000" 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.182655+0000" 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:46.179875+0000" 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.597 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 60, 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.165241+0000" 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.165266+0000" 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.165854+0000" 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.598 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.166383+0000" 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.168243+0000" 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.599 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 62, 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.169955+0000" 2024-04-22T06:16:47.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.169966+0000" 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.170495+0000" 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.601 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.170811+0000" 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.171836+0000" 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 72, 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.184773+0000" 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.603 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.184781+0000" 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.184966+0000" 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.604 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.185336+0000" 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.187271+0000" 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 78, 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.200269+0000" 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.200282+0000" 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.200830+0000" 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.607 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.201112+0000" 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.202177+0000", 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "blocker": { 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.608 INFO:teuthology.orchestra.run.smithi103.stdout: "waiting": 2, 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "held_by_operation_id": 72 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.609 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client_request", 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 79, 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.203000+0000" 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T06:16:47.610 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.203010+0000" 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.203196+0000" 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.203367+0000" 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.611 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-22T06:16:45.204332+0000", 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "blocker": { 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "detail": { 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "waiting": 2, 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "held_by_operation_id": 72 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "blocking_event": { 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T06:16:47.612 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time_event": { 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "0.000000" 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-22T06:16:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-22T06:16:48.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 3427 3411 3.27941 0 - 0.215942 2024-04-22T06:16:49.456 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 3427 3411 3.22972 0 - 0.215942 2024-04-22T06:16:50.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 3427 3411 3.18151 0 - 0.215942 2024-04-22T06:16:51.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 3427 3411 3.13472 0 - 0.215942 2024-04-22T06:16:52.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 3427 3411 3.08929 0 - 0.215942 2024-04-22T06:16:52.615 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 3, 2] 2024-04-22T06:16:52.615 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:16:52.616 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-22T06:16:52.616 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T06:16:52.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T06:16:52.619 INFO:tasks.ceph.osd.1:Started 2024-04-22T06:16:52.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T06:16:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:16:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:16:52.738 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T06:16:53.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 3427 3411 3.04516 0 - 0.215942 2024-04-22T06:16:54.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 3427 3411 3.00227 0 - 0.215942 2024-04-22T06:16:55.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 3427 3411 2.96057 0 - 0.215942 2024-04-22T06:16:56.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 3427 3411 2.92002 0 - 0.215942 2024-04-22T06:16:57.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 3427 3411 2.88056 0 - 0.215942 2024-04-22T06:16:57.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T06:16:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T06:16:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T06:16:57.842 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T06:16:58.457 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 3427 3411 2.84215 0 - 0.215942 2024-04-22T06:16:59.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 3427 3411 2.80475 0 - 0.215942 2024-04-22T06:17:00.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 3427 3411 2.76833 0 - 0.215942 2024-04-22T06:17:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T06:17:01.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 3427 3411 2.73284 0 - 0.215942 2024-04-22T06:17:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T06:17:01.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:17:02.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 3427 3411 2.69825 0 - 0.215942 2024-04-22T06:17:02.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T06:17:03.014 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-22T06:17:03.015 INFO:teuthology.orchestra.run.smithi028.stdout: "ops_in_flight": [] 2024-04-22T06:17:03.015 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-22T06:17:03.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:17:03.459392+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.358454 2024-04-22T06:17:03.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:17:03.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 3583 3567 2.78638 0.573529 0.0885491 0.358454 2024-04-22T06:17:03.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:17:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:03.633+0000 7f73e246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:03.641+0000 7f73e246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 3588 3572 2.75584 0.3125 0.0269987 0.358066 2024-04-22T06:17:05.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 3599 3583 2.73062 0.6875 0.0586531 0.36018 2024-04-22T06:17:05.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:17:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:05.899+0000 7f8f1e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:05.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:05.906+0000 7f8f1e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 3599 3583 2.69772 0 - 0.36018 2024-04-22T06:17:07.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 3599 3583 2.6656 0 - 0.36018 2024-04-22T06:17:08.026 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1] 2024-04-22T06:17:08.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:17:08.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:17:08.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:08.144+0000 7f28601e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:08.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:08.152+0000 7f28601e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:08.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:17:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:17:08.360 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-22T06:17:08.271712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":230687264,"num_objects":3513,"num_object_clones":0,"num_object_copies":10539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3648,"num_write_kb":226598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3604,"ondisk_log_size":3604,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":188744,"kb_used_data":43112,"kb_used_omap":0,"kb_used_meta":145600,"kb_avail":377298616,"statfs":{"total":386547056640,"available":386353782784,"internally_reserved":0,"allocated":44146688,"data_stored":690673357,"data_compressed":32784181,"data_compressed_allocated":43159552,"data_compressed_original":690307072,"omap_allocated":0,"internal_metadata":149094400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001384"},"pg_stats":[{"pgid":"3.f","version":"24'220","reported_seq":255,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-22T06:16:46.079359+0000","last_change":"2024-04-22T06:16:22.549750+0000","last_active":"2024-04-22T06:16:46.079359+0000","last_peered":"2024-04-22T06:16:46.079359+0000","last_clean":"2024-04-22T06:16:46.079359+0000","last_became_active":"2024-04-22T06:16:22.529651+0000","last_became_peered":"2024-04-22T06:16:22.529651+0000","last_unstale":"2024-04-22T06:16:46.079359+0000","last_undegraded":"2024-04-22T06:16:46.079359+0000","last_fullsized":"2024-04-22T06:16:46.079359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":220,"num_write_kb":14080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'205","reported_seq":259,"reported_epoch":31,"state":"active+recovering+degraded","last_fresh":"2024-04-22T06:17:04.652811+0000","last_change":"2024-04-22T06:17:04.555863+0000","last_active":"2024-04-22T06:17:04.652811+0000","last_peered":"2024-04-22T06:17:04.652811+0000","last_clean":"2024-04-22T06:16:27.092529+0000","last_became_active":"2024-04-22T06:17:04.485782+0000","last_became_peered":"2024-04-22T06:17:04.485782+0000","last_unstale":"2024-04-22T06:17:04.652811+0000","last_undegraded":"2024-04-22T06:17:04.368386+0000","last_fullsized":"2024-04-22T06:17:04.652811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":202},{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'222","reported_seq":248,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:27.031948+0000","last_change":"2024-04-22T06:16:22.576738+0000","last_active":"2024-04-22T06:16:27.031948+0000","last_peered":"2024-04-22T06:16:27.031948+0000","last_clean":"2024-04-22T06:16:27.031948+0000","last_became_active":"2024-04-22T06:16:22.567724+0000","last_became_peered":"2024-04-22T06:16:22.567724+0000","last_unstale":"2024-04-22T06:17:01.329807+0000","last_undegraded":"2024-04-22T06:16:27.031948+0000","last_fullsized":"2024-04-22T06:16:27.031948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":14272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'216","reported_seq":251,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-22T06:16:46.135316+0000","last_change":"2024-04-22T06:16:46.136576+0000","last_active":"2024-04-22T06:16:46.135316+0000","last_peered":"2024-04-22T06:16:46.135316+0000","last_clean":"2024-04-22T06:16:27.070079+0000","last_became_active":"2024-04-22T06:16:46.131788+0000","last_became_peered":"2024-04-22T06:16:46.131788+0000","last_unstale":"2024-04-22T06:16:46.135316+0000","last_undegraded":"2024-04-22T06:16:46.135316+0000","last_fullsized":"2024-04-22T06:16:46.135316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"30'216","reported_seq":264,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.382426+0000","last_change":"2024-04-22T06:16:46.239856+0000","last_active":"2024-04-22T06:17:04.382426+0000","last_peered":"2024-04-22T06:17:04.382426+0000","last_clean":"2024-04-22T06:17:04.382426+0000","last_became_active":"2024-04-22T06:16:46.136122+0000","last_became_peered":"2024-04-22T06:16:46.136122+0000","last_unstale":"2024-04-22T06:17:04.382426+0000","last_undegraded":"2024-04-22T06:17:04.382426+0000","last_fullsized":"2024-04-22T06:17:04.382426+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'214","reported_seq":256,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.107253+0000","last_change":"2024-04-22T06:16:44.107943+0000","last_active":"2024-04-22T06:16:44.107943+0000","last_peered":"2024-04-22T06:16:26.961019+0000","last_clean":"2024-04-22T06:16:26.961019+0000","last_became_active":"2024-04-22T06:16:22.545666+0000","last_became_peered":"2024-04-22T06:16:22.545666+0000","last_unstale":"2024-04-22T06:16:45.107253+0000","last_undegraded":"2024-04-22T06:16:45.107253+0000","last_fullsized":"2024-04-22T06:16:45.107253+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":214,"num_write_kb":13696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"24'205","reported_seq":242,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.103376+0000","last_change":"2024-04-22T06:16:44.105498+0000","last_active":"2024-04-22T06:16:44.105498+0000","last_peered":"2024-04-22T06:16:27.015844+0000","last_clean":"2024-04-22T06:16:27.015844+0000","last_became_active":"2024-04-22T06:16:22.519166+0000","last_became_peered":"2024-04-22T06:16:22.519166+0000","last_unstale":"2024-04-22T06:16:45.103376+0000","last_undegraded":"2024-04-22T06:16:45.103376+0000","last_fullsized":"2024-04-22T06:16:45.103376+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'224","reported_seq":250,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:27.085902+0000","last_change":"2024-04-22T06:16:22.581319+0000","last_active":"2024-04-22T06:16:27.085902+0000","last_peered":"2024-04-22T06:16:27.085902+0000","last_clean":"2024-04-22T06:16:27.085902+0000","last_became_active":"2024-04-22T06:16:22.570566+0000","last_became_peered":"2024-04-22T06:16:22.570566+0000","last_unstale":"2024-04-22T06:17:01.329810+0000","last_undegraded":"2024-04-22T06:16:27.085902+0000","last_fullsized":"2024-04-22T06:16:27.085902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":182,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.389440+0000","last_change":"2024-04-22T06:16:46.162663+0000","last_active":"2024-04-22T06:17:04.389440+0000","last_peered":"2024-04-22T06:17:04.389440+0000","last_clean":"2024-04-22T06:17:04.389440+0000","last_became_active":"2024-04-22T06:16:46.156040+0000","last_became_peered":"2024-04-22T06:16:46.156040+0000","last_unstale":"2024-04-22T06:17:04.389440+0000","last_undegraded":"2024-04-22T06:17:04.389440+0000","last_fullsized":"2024-04-22T06:17:04.389440+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:22.651765+0000","last_change":"2024-04-22T06:16:22.653285+0000","last_active":"2024-04-22T06:16:22.651765+0000","last_peered":"2024-04-22T06:16:22.651765+0000","last_clean":"2024-04-22T06:15:30.446240+0000","last_became_active":"2024-04-22T06:16:22.620200+0000","last_became_peered":"2024-04-22T06:16:22.620200+0000","last_unstale":"2024-04-22T06:17:01.329813+0000","last_undegraded":"2024-04-22T06:16:22.651765+0000","last_fullsized":"2024-04-22T06:16:22.651765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'218","reported_seq":244,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:26.753192+0000","last_change":"2024-04-22T06:16:22.540991+0000","last_active":"2024-04-22T06:16:26.753192+0000","last_peered":"2024-04-22T06:16:26.753192+0000","last_clean":"2024-04-22T06:16:26.753192+0000","last_became_active":"2024-04-22T06:16:22.531621+0000","last_became_peered":"2024-04-22T06:16:22.531621+0000","last_unstale":"2024-04-22T06:17:01.329819+0000","last_undegraded":"2024-04-22T06:16:26.753192+0000","last_fullsized":"2024-04-22T06:16:26.753192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.100701+0000","last_change":"2024-04-22T06:16:44.108908+0000","last_active":"2024-04-22T06:16:44.108908+0000","last_peered":"2024-04-22T06:16:22.507973+0000","last_clean":"2024-04-22T06:16:22.507973+0000","last_became_active":"2024-04-22T06:16:22.502317+0000","last_became_peered":"2024-04-22T06:16:22.502317+0000","last_unstale":"2024-04-22T06:16:45.100701+0000","last_undegraded":"2024-04-22T06:16:45.100701+0000","last_fullsized":"2024-04-22T06:16:45.100701+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"29'245","reported_seq":287,"reported_epoch":31,"state":"activating","last_fresh":"2024-04-22T06:17:04.354251+0000","last_change":"2024-04-22T06:17:04.361820+0000","last_active":"2024-04-22T06:17:03.439796+0000","last_peered":"2024-04-22T06:17:03.240245+0000","last_clean":"2024-04-22T06:16:46.082243+0000","last_became_active":"2024-04-22T06:17:02.410113+0000","last_became_peered":"2024-04-22T06:17:02.410113+0000","last_unstale":"2024-04-22T06:17:04.354251+0000","last_undegraded":"2024-04-22T06:17:04.354251+0000","last_fullsized":"2024-04-22T06:17:04.354251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":15680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,1,3","objects":230},{"shards":"0,3","objects":15}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:22.632730+0000","last_change":"2024-04-22T06:16:22.634721+0000","last_active":"2024-04-22T06:16:22.632730+0000","last_peered":"2024-04-22T06:16:22.632730+0000","last_clean":"2024-04-22T06:15:51.511180+0000","last_became_active":"2024-04-22T06:16:22.626588+0000","last_became_peered":"2024-04-22T06:16:22.626588+0000","last_unstale":"2024-04-22T06:17:01.329821+0000","last_undegraded":"2024-04-22T06:16:22.632730+0000","last_fullsized":"2024-04-22T06:16:22.632730+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"29'226","reported_seq":274,"reported_epoch":31,"state":"activating","last_fresh":"2024-04-22T06:17:04.394989+0000","last_change":"2024-04-22T06:17:04.413509+0000","last_active":"2024-04-22T06:17:03.448318+0000","last_peered":"2024-04-22T06:17:03.414407+0000","last_clean":"2024-04-22T06:16:27.121820+0000","last_became_active":"2024-04-22T06:17:02.450995+0000","last_became_peered":"2024-04-22T06:17:02.450995+0000","last_unstale":"2024-04-22T06:17:04.394989+0000","last_undegraded":"2024-04-22T06:17:04.394989+0000","last_fullsized":"2024-04-22T06:17:04.394989+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":226,"num_write_kb":14464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":["3"],"object_location_counts":[{"shards":"1,2,3","objects":217},{"shards":"2,3","objects":8},{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.101076+0000","last_change":"2024-04-22T06:16:44.110391+0000","last_active":"2024-04-22T06:16:44.110391+0000","last_peered":"2024-04-22T06:16:22.556101+0000","last_clean":"2024-04-22T06:16:22.556101+0000","last_became_active":"2024-04-22T06:16:22.539404+0000","last_became_peered":"2024-04-22T06:16:22.539404+0000","last_unstale":"2024-04-22T06:16:45.101076+0000","last_undegraded":"2024-04-22T06:16:45.101076+0000","last_fullsized":"2024-04-22T06:16:45.101076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"30'207","reported_seq":253,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.357576+0000","last_change":"2024-04-22T06:16:46.167477+0000","last_active":"2024-04-22T06:17:04.357576+0000","last_peered":"2024-04-22T06:17:04.357576+0000","last_clean":"2024-04-22T06:17:04.357576+0000","last_became_active":"2024-04-22T06:16:46.158373+0000","last_became_peered":"2024-04-22T06:16:46.158373+0000","last_unstale":"2024-04-22T06:17:04.357576+0000","last_undegraded":"2024-04-22T06:17:04.357576+0000","last_fullsized":"2024-04-22T06:17:04.357576+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13565952,"num_objects":207,"num_object_clones":0,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":207,"num_write_kb":13248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"24'216","reported_seq":242,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:27.138171+0000","last_change":"2024-04-22T06:16:22.649607+0000","last_active":"2024-04-22T06:16:27.138171+0000","last_peered":"2024-04-22T06:16:27.138171+0000","last_clean":"2024-04-22T06:16:27.138171+0000","last_became_active":"2024-04-22T06:16:22.614839+0000","last_became_peered":"2024-04-22T06:16:22.614839+0000","last_unstale":"2024-04-22T06:17:01.329824+0000","last_undegraded":"2024-04-22T06:16:27.138171+0000","last_fullsized":"2024-04-22T06:16:27.138171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":13888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.358744+0000","last_change":"2024-04-22T06:16:46.184804+0000","last_active":"2024-04-22T06:17:04.358744+0000","last_peered":"2024-04-22T06:17:04.358744+0000","last_clean":"2024-04-22T06:17:04.358744+0000","last_became_active":"2024-04-22T06:16:46.174487+0000","last_became_peered":"2024-04-22T06:16:46.174487+0000","last_unstale":"2024-04-22T06:17:04.358744+0000","last_undegraded":"2024-04-22T06:17:04.358744+0000","last_fullsized":"2024-04-22T06:17:04.358744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'216","reported_seq":262,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.378153+0000","last_change":"2024-04-22T06:16:46.194179+0000","last_active":"2024-04-22T06:17:04.378153+0000","last_peered":"2024-04-22T06:17:04.378153+0000","last_clean":"2024-04-22T06:17:04.378153+0000","last_became_active":"2024-04-22T06:16:46.189232+0000","last_became_peered":"2024-04-22T06:16:46.189232+0000","last_unstale":"2024-04-22T06:17:04.378153+0000","last_undegraded":"2024-04-22T06:17:04.378153+0000","last_fullsized":"2024-04-22T06:17:04.378153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.416545+0000","last_change":"2024-04-22T06:16:46.153214+0000","last_active":"2024-04-22T06:17:04.416545+0000","last_peered":"2024-04-22T06:17:04.416545+0000","last_clean":"2024-04-22T06:17:04.416545+0000","last_became_active":"2024-04-22T06:16:46.147164+0000","last_became_peered":"2024-04-22T06:16:46.147164+0000","last_unstale":"2024-04-22T06:17:04.416545+0000","last_undegraded":"2024-04-22T06:17:04.416545+0000","last_fullsized":"2024-04-22T06:17:04.416545+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"29'240","reported_seq":282,"reported_epoch":31,"state":"activating","last_fresh":"2024-04-22T06:17:04.363781+0000","last_change":"2024-04-22T06:17:04.380141+0000","last_active":"2024-04-22T06:17:03.435883+0000","last_peered":"2024-04-22T06:17:03.398532+0000","last_clean":"2024-04-22T06:16:46.067917+0000","last_became_active":"2024-04-22T06:17:02.404755+0000","last_became_peered":"2024-04-22T06:17:02.404755+0000","last_unstale":"2024-04-22T06:17:04.363781+0000","last_undegraded":"2024-04-22T06:17:04.363781+0000","last_fullsized":"2024-04-22T06:17:04.363781+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":15360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,1,3","objects":226},{"shards":"0,3","objects":12},{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.372723+0000","last_change":"2024-04-22T06:16:46.204230+0000","last_active":"2024-04-22T06:17:04.372723+0000","last_peered":"2024-04-22T06:17:04.372723+0000","last_clean":"2024-04-22T06:17:04.372723+0000","last_became_active":"2024-04-22T06:16:46.199731+0000","last_became_peered":"2024-04-22T06:16:46.199731+0000","last_unstale":"2024-04-22T06:17:04.372723+0000","last_undegraded":"2024-04-22T06:17:04.372723+0000","last_fullsized":"2024-04-22T06:17:04.372723+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'217","reported_seq":275,"reported_epoch":31,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T06:17:04.653243+0000","last_change":"2024-04-22T06:17:04.653243+0000","last_active":"2024-04-22T06:17:04.653243+0000","last_peered":"2024-04-22T06:17:04.653243+0000","last_clean":"2024-04-22T06:16:27.058758+0000","last_became_active":"2024-04-22T06:17:04.510969+0000","last_became_peered":"2024-04-22T06:17:04.510969+0000","last_unstale":"2024-04-22T06:17:04.653243+0000","last_undegraded":"2024-04-22T06:17:04.389111+0000","last_fullsized":"2024-04-22T06:17:04.653243+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":13888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":207},{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:22.583303+0000","last_change":"2024-04-22T06:16:22.584160+0000","last_active":"2024-04-22T06:16:22.583303+0000","last_peered":"2024-04-22T06:16:22.583303+0000","last_clean":"2024-04-22T06:15:30.460326+0000","last_became_active":"2024-04-22T06:16:22.576799+0000","last_became_peered":"2024-04-22T06:16:22.576799+0000","last_unstale":"2024-04-22T06:17:01.329827+0000","last_undegraded":"2024-04-22T06:16:22.583303+0000","last_fullsized":"2024-04-22T06:16:22.583303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":10533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3511,"num_write_kb":224704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43081728,"data_stored":689307648,"data_compressed":32728686,"data_compressed_allocated":43081728,"data_compressed_original":689307648,"omap_allocated":0,"internal_metadata":0},"log_size":3507,"ondisk_log_size":3507,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59818,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378697,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52224,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":94319616,"statfs":{"total":96636764160,"available":96583286784,"internally_reserved":0,"allocated":13164544,"data_stored":206329268,"data_compressed":9794383,"data_compressed_allocated":12894208,"data_compressed_original":206225408,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149715,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45540,"kb_used_data":9948,"kb_used_omap":0,"kb_used_meta":35584,"kb_avail":94326300,"statfs":{"total":96636764160,"available":96590131200,"internally_reserved":0,"allocated":10186752,"data_stored":158684596,"data_compressed":7532210,"data_compressed_allocated":9916416,"data_compressed_original":158580736,"omap_allocated":0,"internal_metadata":36438016},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":46124,"kb_used_data":10596,"kb_used_omap":0,"kb_used_meta":35520,"kb_avail":94325716,"statfs":{"total":96636764160,"available":96589533184,"internally_reserved":0,"allocated":10850304,"data_stored":170459000,"data_compressed":8090363,"data_compressed_allocated":10649600,"data_compressed_original":170393600,"omap_allocated":0,"internal_metadata":36372480},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44856,"kb_used_data":9712,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94326984,"statfs":{"total":96636764160,"available":96590831616,"internally_reserved":0,"allocated":9945088,"data_stored":155200493,"data_compressed":7367225,"data_compressed_allocated":9699328,"data_compressed_original":155107328,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9658368,"data_stored":154533888,"data_compressed":7337316,"data_compressed_allocated":9658368,"data_compressed_original":154533888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10649600,"data_stored":170393600,"data_compressed":8090363,"data_compressed_allocated":10649600,"data_compressed_original":170393600,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9900032,"data_stored":158400512,"data_compressed":7520973,"data_compressed_allocated":9900032,"data_compressed_original":158400512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12873728,"data_stored":205979648,"data_compressed":9780034,"data_compressed_allocated":12873728,"data_compressed_original":205979648,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:17:08.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.e 3.6 2024-04-22T06:17:08.458 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 3599 3583 2.63424 0 - 0.36018 2024-04-22T06:17:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:08.468+0000 7feef9ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:08.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:08.479+0000 7feef9ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:instructing pg(s) [3.6,3.e] on osd.2 to force-backfill; 2024-04-22T06:17:08.671 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T06:17:09.459 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 3599 3583 2.60361 0 - 0.36018 2024-04-22T06:17:10.459 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 3599 3583 2.57368 0 - 0.36018 2024-04-22T06:17:10.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-22T06:17:10.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-22T06:17:10.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-22T06:17:10.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-22T06:17:11.459 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 3599 3583 2.54444 0 - 0.36018 2024-04-22T06:17:12.459 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 3599 3583 2.51585 0 - 0.36018 2024-04-22T06:17:13.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 90 16 3599 3583 2.48789 0 - 0.36018 2024-04-22T06:17:13.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1] 2024-04-22T06:17:13.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:17:13.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:17:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:13.807+0000 7fae15bcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:13.816+0000 7fae15bcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-22T06:17:14.011 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-22T06:17:14.023 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-22T06:17:12.272817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":230687264,"num_objects":3513,"num_object_clones":0,"num_object_copies":10539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3648,"num_write_kb":226598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3604,"ondisk_log_size":3604,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":188744,"kb_used_data":43112,"kb_used_omap":0,"kb_used_meta":145600,"kb_avail":377298616,"statfs":{"total":386547056640,"available":386353782784,"internally_reserved":0,"allocated":44146688,"data_stored":690673357,"data_compressed":32784181,"data_compressed_allocated":43159552,"data_compressed_original":690307072,"omap_allocated":0,"internal_metadata":149094400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.939781"},"pg_stats":[{"pgid":"3.f","version":"24'220","reported_seq":255,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-22T06:16:46.079359+0000","last_change":"2024-04-22T06:16:22.549750+0000","last_active":"2024-04-22T06:16:46.079359+0000","last_peered":"2024-04-22T06:16:46.079359+0000","last_clean":"2024-04-22T06:16:46.079359+0000","last_became_active":"2024-04-22T06:16:22.529651+0000","last_became_peered":"2024-04-22T06:16:22.529651+0000","last_unstale":"2024-04-22T06:16:46.079359+0000","last_undegraded":"2024-04-22T06:16:46.079359+0000","last_fullsized":"2024-04-22T06:16:46.079359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":220,"num_write_kb":14080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'205","reported_seq":259,"reported_epoch":31,"state":"active+recovering+degraded","last_fresh":"2024-04-22T06:17:04.652811+0000","last_change":"2024-04-22T06:17:04.555863+0000","last_active":"2024-04-22T06:17:04.652811+0000","last_peered":"2024-04-22T06:17:04.652811+0000","last_clean":"2024-04-22T06:16:27.092529+0000","last_became_active":"2024-04-22T06:17:04.485782+0000","last_became_peered":"2024-04-22T06:17:04.485782+0000","last_unstale":"2024-04-22T06:17:04.652811+0000","last_undegraded":"2024-04-22T06:17:04.368386+0000","last_fullsized":"2024-04-22T06:17:04.652811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":202},{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'222","reported_seq":248,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:27.031948+0000","last_change":"2024-04-22T06:16:22.576738+0000","last_active":"2024-04-22T06:16:27.031948+0000","last_peered":"2024-04-22T06:16:27.031948+0000","last_clean":"2024-04-22T06:16:27.031948+0000","last_became_active":"2024-04-22T06:16:22.567724+0000","last_became_peered":"2024-04-22T06:16:22.567724+0000","last_unstale":"2024-04-22T06:17:01.329807+0000","last_undegraded":"2024-04-22T06:16:27.031948+0000","last_fullsized":"2024-04-22T06:16:27.031948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":14272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'216","reported_seq":251,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-22T06:16:46.135316+0000","last_change":"2024-04-22T06:16:46.136576+0000","last_active":"2024-04-22T06:16:46.135316+0000","last_peered":"2024-04-22T06:16:46.135316+0000","last_clean":"2024-04-22T06:16:27.070079+0000","last_became_active":"2024-04-22T06:16:46.131788+0000","last_became_peered":"2024-04-22T06:16:46.131788+0000","last_unstale":"2024-04-22T06:16:46.135316+0000","last_undegraded":"2024-04-22T06:16:46.135316+0000","last_fullsized":"2024-04-22T06:16:46.135316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"30'216","reported_seq":264,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.382426+0000","last_change":"2024-04-22T06:16:46.239856+0000","last_active":"2024-04-22T06:17:04.382426+0000","last_peered":"2024-04-22T06:17:04.382426+0000","last_clean":"2024-04-22T06:17:04.382426+0000","last_became_active":"2024-04-22T06:16:46.136122+0000","last_became_peered":"2024-04-22T06:16:46.136122+0000","last_unstale":"2024-04-22T06:17:04.382426+0000","last_undegraded":"2024-04-22T06:17:04.382426+0000","last_fullsized":"2024-04-22T06:17:04.382426+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'214","reported_seq":256,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.107253+0000","last_change":"2024-04-22T06:16:44.107943+0000","last_active":"2024-04-22T06:16:44.107943+0000","last_peered":"2024-04-22T06:16:26.961019+0000","last_clean":"2024-04-22T06:16:26.961019+0000","last_became_active":"2024-04-22T06:16:22.545666+0000","last_became_peered":"2024-04-22T06:16:22.545666+0000","last_unstale":"2024-04-22T06:16:45.107253+0000","last_undegraded":"2024-04-22T06:16:45.107253+0000","last_fullsized":"2024-04-22T06:16:45.107253+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":214,"num_write_kb":13696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"24'205","reported_seq":242,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.103376+0000","last_change":"2024-04-22T06:16:44.105498+0000","last_active":"2024-04-22T06:16:44.105498+0000","last_peered":"2024-04-22T06:16:27.015844+0000","last_clean":"2024-04-22T06:16:27.015844+0000","last_became_active":"2024-04-22T06:16:22.519166+0000","last_became_peered":"2024-04-22T06:16:22.519166+0000","last_unstale":"2024-04-22T06:16:45.103376+0000","last_undegraded":"2024-04-22T06:16:45.103376+0000","last_fullsized":"2024-04-22T06:16:45.103376+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'224","reported_seq":250,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:27.085902+0000","last_change":"2024-04-22T06:16:22.581319+0000","last_active":"2024-04-22T06:16:27.085902+0000","last_peered":"2024-04-22T06:16:27.085902+0000","last_clean":"2024-04-22T06:16:27.085902+0000","last_became_active":"2024-04-22T06:16:22.570566+0000","last_became_peered":"2024-04-22T06:16:22.570566+0000","last_unstale":"2024-04-22T06:17:01.329810+0000","last_undegraded":"2024-04-22T06:16:27.085902+0000","last_fullsized":"2024-04-22T06:16:27.085902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":182,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.389440+0000","last_change":"2024-04-22T06:16:46.162663+0000","last_active":"2024-04-22T06:17:04.389440+0000","last_peered":"2024-04-22T06:17:04.389440+0000","last_clean":"2024-04-22T06:17:04.389440+0000","last_became_active":"2024-04-22T06:16:46.156040+0000","last_became_peered":"2024-04-22T06:16:46.156040+0000","last_unstale":"2024-04-22T06:17:04.389440+0000","last_undegraded":"2024-04-22T06:17:04.389440+0000","last_fullsized":"2024-04-22T06:17:04.389440+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:27.257222+0000","last_clean_scrub_stamp":"2024-04-22T06:15:27.257222+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:22.651765+0000","last_change":"2024-04-22T06:16:22.653285+0000","last_active":"2024-04-22T06:16:22.651765+0000","last_peered":"2024-04-22T06:16:22.651765+0000","last_clean":"2024-04-22T06:15:30.446240+0000","last_became_active":"2024-04-22T06:16:22.620200+0000","last_became_peered":"2024-04-22T06:16:22.620200+0000","last_unstale":"2024-04-22T06:17:01.329813+0000","last_undegraded":"2024-04-22T06:16:22.651765+0000","last_fullsized":"2024-04-22T06:16:22.651765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'218","reported_seq":244,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:26.753192+0000","last_change":"2024-04-22T06:16:22.540991+0000","last_active":"2024-04-22T06:16:26.753192+0000","last_peered":"2024-04-22T06:16:26.753192+0000","last_clean":"2024-04-22T06:16:26.753192+0000","last_became_active":"2024-04-22T06:16:22.531621+0000","last_became_peered":"2024-04-22T06:16:22.531621+0000","last_unstale":"2024-04-22T06:17:01.329819+0000","last_undegraded":"2024-04-22T06:16:26.753192+0000","last_fullsized":"2024-04-22T06:16:26.753192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.100701+0000","last_change":"2024-04-22T06:16:44.108908+0000","last_active":"2024-04-22T06:16:44.108908+0000","last_peered":"2024-04-22T06:16:22.507973+0000","last_clean":"2024-04-22T06:16:22.507973+0000","last_became_active":"2024-04-22T06:16:22.502317+0000","last_became_peered":"2024-04-22T06:16:22.502317+0000","last_unstale":"2024-04-22T06:16:45.100701+0000","last_undegraded":"2024-04-22T06:16:45.100701+0000","last_fullsized":"2024-04-22T06:16:45.100701+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"29'245","reported_seq":287,"reported_epoch":31,"state":"activating","last_fresh":"2024-04-22T06:17:04.354251+0000","last_change":"2024-04-22T06:17:04.361820+0000","last_active":"2024-04-22T06:17:03.439796+0000","last_peered":"2024-04-22T06:17:03.240245+0000","last_clean":"2024-04-22T06:16:46.082243+0000","last_became_active":"2024-04-22T06:17:02.410113+0000","last_became_peered":"2024-04-22T06:17:02.410113+0000","last_unstale":"2024-04-22T06:17:04.354251+0000","last_undegraded":"2024-04-22T06:17:04.354251+0000","last_fullsized":"2024-04-22T06:17:04.354251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":15680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,1,3","objects":230},{"shards":"0,3","objects":15}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:22.632730+0000","last_change":"2024-04-22T06:16:22.634721+0000","last_active":"2024-04-22T06:16:22.632730+0000","last_peered":"2024-04-22T06:16:22.632730+0000","last_clean":"2024-04-22T06:15:51.511180+0000","last_became_active":"2024-04-22T06:16:22.626588+0000","last_became_peered":"2024-04-22T06:16:22.626588+0000","last_unstale":"2024-04-22T06:17:01.329821+0000","last_undegraded":"2024-04-22T06:16:22.632730+0000","last_fullsized":"2024-04-22T06:16:22.632730+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"29'226","reported_seq":274,"reported_epoch":31,"state":"activating","last_fresh":"2024-04-22T06:17:04.394989+0000","last_change":"2024-04-22T06:17:04.413509+0000","last_active":"2024-04-22T06:17:03.448318+0000","last_peered":"2024-04-22T06:17:03.414407+0000","last_clean":"2024-04-22T06:16:27.121820+0000","last_became_active":"2024-04-22T06:17:02.450995+0000","last_became_peered":"2024-04-22T06:17:02.450995+0000","last_unstale":"2024-04-22T06:17:04.394989+0000","last_undegraded":"2024-04-22T06:17:04.394989+0000","last_fullsized":"2024-04-22T06:17:04.394989+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":226,"num_write_kb":14464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":["3"],"object_location_counts":[{"shards":"1,2,3","objects":217},{"shards":"2,3","objects":8},{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T06:16:45.101076+0000","last_change":"2024-04-22T06:16:44.110391+0000","last_active":"2024-04-22T06:16:44.110391+0000","last_peered":"2024-04-22T06:16:22.556101+0000","last_clean":"2024-04-22T06:16:22.556101+0000","last_became_active":"2024-04-22T06:16:22.539404+0000","last_became_peered":"2024-04-22T06:16:22.539404+0000","last_unstale":"2024-04-22T06:16:45.101076+0000","last_undegraded":"2024-04-22T06:16:45.101076+0000","last_fullsized":"2024-04-22T06:16:45.101076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"30'207","reported_seq":253,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.357576+0000","last_change":"2024-04-22T06:16:46.167477+0000","last_active":"2024-04-22T06:17:04.357576+0000","last_peered":"2024-04-22T06:17:04.357576+0000","last_clean":"2024-04-22T06:17:04.357576+0000","last_became_active":"2024-04-22T06:16:46.158373+0000","last_became_peered":"2024-04-22T06:16:46.158373+0000","last_unstale":"2024-04-22T06:17:04.357576+0000","last_undegraded":"2024-04-22T06:17:04.357576+0000","last_fullsized":"2024-04-22T06:17:04.357576+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13565952,"num_objects":207,"num_object_clones":0,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":207,"num_write_kb":13248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"24'216","reported_seq":242,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:27.138171+0000","last_change":"2024-04-22T06:16:22.649607+0000","last_active":"2024-04-22T06:16:27.138171+0000","last_peered":"2024-04-22T06:16:27.138171+0000","last_clean":"2024-04-22T06:16:27.138171+0000","last_became_active":"2024-04-22T06:16:22.614839+0000","last_became_peered":"2024-04-22T06:16:22.614839+0000","last_unstale":"2024-04-22T06:17:01.329824+0000","last_undegraded":"2024-04-22T06:16:27.138171+0000","last_fullsized":"2024-04-22T06:16:27.138171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":13888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.358744+0000","last_change":"2024-04-22T06:16:46.184804+0000","last_active":"2024-04-22T06:17:04.358744+0000","last_peered":"2024-04-22T06:17:04.358744+0000","last_clean":"2024-04-22T06:17:04.358744+0000","last_became_active":"2024-04-22T06:16:46.174487+0000","last_became_peered":"2024-04-22T06:16:46.174487+0000","last_unstale":"2024-04-22T06:17:04.358744+0000","last_undegraded":"2024-04-22T06:17:04.358744+0000","last_fullsized":"2024-04-22T06:17:04.358744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'216","reported_seq":262,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.378153+0000","last_change":"2024-04-22T06:16:46.194179+0000","last_active":"2024-04-22T06:17:04.378153+0000","last_peered":"2024-04-22T06:17:04.378153+0000","last_clean":"2024-04-22T06:17:04.378153+0000","last_became_active":"2024-04-22T06:16:46.189232+0000","last_became_peered":"2024-04-22T06:16:46.189232+0000","last_unstale":"2024-04-22T06:17:04.378153+0000","last_undegraded":"2024-04-22T06:17:04.378153+0000","last_fullsized":"2024-04-22T06:17:04.378153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.416545+0000","last_change":"2024-04-22T06:16:46.153214+0000","last_active":"2024-04-22T06:17:04.416545+0000","last_peered":"2024-04-22T06:17:04.416545+0000","last_clean":"2024-04-22T06:17:04.416545+0000","last_became_active":"2024-04-22T06:16:46.147164+0000","last_became_peered":"2024-04-22T06:16:46.147164+0000","last_unstale":"2024-04-22T06:17:04.416545+0000","last_undegraded":"2024-04-22T06:17:04.416545+0000","last_fullsized":"2024-04-22T06:17:04.416545+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"29'240","reported_seq":282,"reported_epoch":31,"state":"activating","last_fresh":"2024-04-22T06:17:04.363781+0000","last_change":"2024-04-22T06:17:04.380141+0000","last_active":"2024-04-22T06:17:03.435883+0000","last_peered":"2024-04-22T06:17:03.398532+0000","last_clean":"2024-04-22T06:16:46.067917+0000","last_became_active":"2024-04-22T06:17:02.404755+0000","last_became_peered":"2024-04-22T06:17:02.404755+0000","last_unstale":"2024-04-22T06:17:04.363781+0000","last_undegraded":"2024-04-22T06:17:04.363781+0000","last_fullsized":"2024-04-22T06:17:04.363781+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":15360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,1,3","objects":226},{"shards":"0,3","objects":12},{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T06:17:04.372723+0000","last_change":"2024-04-22T06:16:46.204230+0000","last_active":"2024-04-22T06:17:04.372723+0000","last_peered":"2024-04-22T06:17:04.372723+0000","last_clean":"2024-04-22T06:17:04.372723+0000","last_became_active":"2024-04-22T06:16:46.199731+0000","last_became_peered":"2024-04-22T06:16:46.199731+0000","last_unstale":"2024-04-22T06:17:04.372723+0000","last_undegraded":"2024-04-22T06:17:04.372723+0000","last_fullsized":"2024-04-22T06:17:04.372723+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'217","reported_seq":275,"reported_epoch":31,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T06:17:04.653243+0000","last_change":"2024-04-22T06:17:04.653243+0000","last_active":"2024-04-22T06:17:04.653243+0000","last_peered":"2024-04-22T06:17:04.653243+0000","last_clean":"2024-04-22T06:16:27.058758+0000","last_became_active":"2024-04-22T06:17:04.510969+0000","last_became_peered":"2024-04-22T06:17:04.510969+0000","last_unstale":"2024-04-22T06:17:04.653243+0000","last_undegraded":"2024-04-22T06:17:04.389111+0000","last_fullsized":"2024-04-22T06:17:04.653243+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:41.344834+0000","last_clean_scrub_stamp":"2024-04-22T06:15:41.344834+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":13888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":207},{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"stale+active+clean","last_fresh":"2024-04-22T06:16:22.583303+0000","last_change":"2024-04-22T06:16:22.584160+0000","last_active":"2024-04-22T06:16:22.583303+0000","last_peered":"2024-04-22T06:16:22.583303+0000","last_clean":"2024-04-22T06:15:30.460326+0000","last_became_active":"2024-04-22T06:16:22.576799+0000","last_became_peered":"2024-04-22T06:16:22.576799+0000","last_unstale":"2024-04-22T06:17:01.329827+0000","last_undegraded":"2024-04-22T06:16:22.583303+0000","last_fullsized":"2024-04-22T06:16:22.583303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:15:29.310548+0000","last_clean_scrub_stamp":"2024-04-22T06:15:29.310548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":10533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3511,"num_write_kb":224704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43081728,"data_stored":689307648,"data_compressed":32728686,"data_compressed_allocated":43081728,"data_compressed_original":689307648,"omap_allocated":0,"internal_metadata":0},"log_size":3507,"ondisk_log_size":3507,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59818,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378697,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52224,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":94319616,"statfs":{"total":96636764160,"available":96583286784,"internally_reserved":0,"allocated":13164544,"data_stored":206329268,"data_compressed":9794383,"data_compressed_allocated":12894208,"data_compressed_original":206225408,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149715,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45540,"kb_used_data":9948,"kb_used_omap":0,"kb_used_meta":35584,"kb_avail":94326300,"statfs":{"total":96636764160,"available":96590131200,"internally_reserved":0,"allocated":10186752,"data_stored":158684596,"data_compressed":7532210,"data_compressed_allocated":9916416,"data_compressed_original":158580736,"omap_allocated":0,"internal_metadata":36438016},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":46124,"kb_used_data":10596,"kb_used_omap":0,"kb_used_meta":35520,"kb_avail":94325716,"statfs":{"total":96636764160,"available":96589533184,"internally_reserved":0,"allocated":10850304,"data_stored":170459000,"data_compressed":8090363,"data_compressed_allocated":10649600,"data_compressed_original":170393600,"omap_allocated":0,"internal_metadata":36372480},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44856,"kb_used_data":9712,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94326984,"statfs":{"total":96636764160,"available":96590831616,"internally_reserved":0,"allocated":9945088,"data_stored":155200493,"data_compressed":7367225,"data_compressed_allocated":9699328,"data_compressed_original":155107328,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9658368,"data_stored":154533888,"data_compressed":7337316,"data_compressed_allocated":9658368,"data_compressed_original":154533888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10649600,"data_stored":170393600,"data_compressed":8090363,"data_compressed_allocated":10649600,"data_compressed_original":170393600,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9900032,"data_stored":158400512,"data_compressed":7520973,"data_compressed_allocated":9900032,"data_compressed_original":158400512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12873728,"data_stored":205979648,"data_compressed":9780034,"data_compressed_allocated":12873728,"data_compressed_original":205979648,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:17:14.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 3.e 2024-04-22T06:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:14.143+0000 7f72c546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:14.152+0000 7f72c546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:14.350 INFO:teuthology.orchestra.run.smithi028.stderr:instructing pg(s) [3.e] on osd.2 to force-recovery; 2024-04-22T06:17:14.351 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T06:17:14.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 91 16 3599 3583 2.46055 0 - 0.36018 2024-04-22T06:17:15.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 92 16 3599 3583 2.43381 0 - 0.36018 2024-04-22T06:17:16.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-22T06:17:16.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-22T06:17:16.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-04-22T06:17:16.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2024-04-22T06:17:16.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 93 16 3599 3583 2.40763 0 - 0.36018 2024-04-22T06:17:17.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 94 16 3599 3583 2.38202 0 - 0.36018 2024-04-22T06:17:18.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 95 16 3599 3583 2.35695 0 - 0.36018 2024-04-22T06:17:19.366 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1] 2024-04-22T06:17:19.366 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:17:19.366 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T06:17:19.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T06:17:19.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 96 16 3599 3583 2.3324 0 - 0.36018 2024-04-22T06:17:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:19.488+0000 7f667f0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:17:19.497+0000 7f667f0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 97 16 3599 3583 2.30835 0 - 0.36018 2024-04-22T06:17:21.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~10s 2024-04-22T06:17:21.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2024-04-22T06:17:21.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~10s 2024-04-22T06:17:21.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~10s 2024-04-22T06:17:21.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 98 16 3599 3583 2.2848 0 - 0.36018 2024-04-22T06:17:22.460 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 99 16 3599 3583 2.26172 0 - 0.36018 2024-04-22T06:17:23.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:17:23.461985+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:17:23.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:17:23.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 100 16 3599 3583 2.2391 0 - 0.36018 2024-04-22T06:17:24.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 101 16 3599 3583 2.21693 0 - 0.36018 2024-04-22T06:17:25.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 102 16 3599 3583 2.19519 0 - 0.36018 2024-04-22T06:17:26.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-04-22T06:17:26.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-22T06:17:26.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2024-04-22T06:17:26.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~16s 2024-04-22T06:17:26.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 103 16 3599 3583 2.17388 0 - 0.36018 2024-04-22T06:17:27.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 104 16 3599 3583 2.15298 0 - 0.36018 2024-04-22T06:17:28.461 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 105 16 3599 3583 2.13247 0 - 0.36018 2024-04-22T06:17:29.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 106 16 3599 3583 2.11236 0 - 0.36018 2024-04-22T06:17:30.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 107 16 3599 3583 2.09261 0 - 0.36018 2024-04-22T06:17:31.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 108 16 3599 3583 2.07324 0 - 0.36018 2024-04-22T06:17:31.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2024-04-22T06:17:31.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2024-04-22T06:17:31.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~21s 2024-04-22T06:17:31.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~21s 2024-04-22T06:17:32.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 109 16 3599 3583 2.05422 0 - 0.36018 2024-04-22T06:17:33.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 110 16 3599 3583 2.03554 0 - 0.36018 2024-04-22T06:17:34.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 111 16 3599 3583 2.0172 0 - 0.36018 2024-04-22T06:17:35.462 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 112 16 3599 3583 1.99919 0 - 0.36018 2024-04-22T06:17:36.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 113 16 3599 3583 1.9815 0 - 0.36018 2024-04-22T06:17:36.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~26s 2024-04-22T06:17:36.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~26s 2024-04-22T06:17:36.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~26s 2024-04-22T06:17:36.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~26s 2024-04-22T06:17:37.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 114 16 3599 3583 1.96412 0 - 0.36018 2024-04-22T06:17:38.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 115 16 3599 3583 1.94704 0 - 0.36018 2024-04-22T06:17:39.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 116 16 3599 3583 1.93025 0 - 0.36018 2024-04-22T06:17:40.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 117 16 3599 3583 1.91375 0 - 0.36018 2024-04-22T06:17:41.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 118 16 3599 3583 1.89754 0 - 0.36018 2024-04-22T06:17:42.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~31s 2024-04-22T06:17:42.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~31s 2024-04-22T06:17:42.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~31s 2024-04-22T06:17:42.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~31s 2024-04-22T06:17:42.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 119 16 3599 3583 1.88159 0 - 0.36018 2024-04-22T06:17:43.463 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:17:43.464885+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:17:43.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:17:43.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 120 16 3599 3583 1.86591 0 - 0.36018 2024-04-22T06:17:44.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 121 16 3599 3583 1.85049 0 - 0.36018 2024-04-22T06:17:45.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 122 16 3599 3583 1.83532 0 - 0.36018 2024-04-22T06:17:46.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 123 16 3599 3583 1.8204 0 - 0.36018 2024-04-22T06:17:47.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~36s 2024-04-22T06:17:47.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~36s 2024-04-22T06:17:47.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~36s 2024-04-22T06:17:47.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~36s 2024-04-22T06:17:47.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 124 16 3599 3583 1.80572 0 - 0.36018 2024-04-22T06:17:48.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 125 16 3599 3583 1.79127 0 - 0.36018 2024-04-22T06:17:49.464 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 126 16 3599 3583 1.77705 0 - 0.36018 2024-04-22T06:17:50.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 127 16 3599 3583 1.76306 0 - 0.36018 2024-04-22T06:17:51.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 128 16 3599 3583 1.74929 0 - 0.36018 2024-04-22T06:17:52.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2024-04-22T06:17:52.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2024-04-22T06:17:52.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~42s 2024-04-22T06:17:52.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~42s 2024-04-22T06:17:52.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 129 16 3599 3583 1.73573 0 - 0.36018 2024-04-22T06:17:53.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 130 16 3599 3583 1.72237 0 - 0.36018 2024-04-22T06:17:54.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 131 16 3599 3583 1.70923 0 - 0.36018 2024-04-22T06:17:55.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 132 16 3599 3583 1.69628 0 - 0.36018 2024-04-22T06:17:56.465 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 133 16 3599 3583 1.68352 0 - 0.36018 2024-04-22T06:17:57.466 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 134 16 3599 3583 1.67096 0 - 0.36018 2024-04-22T06:17:57.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~47s 2024-04-22T06:17:57.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~47s 2024-04-22T06:17:57.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~47s 2024-04-22T06:17:57.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~47s 2024-04-22T06:17:58.466 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 135 16 3599 3583 1.65858 0 - 0.36018 2024-04-22T06:17:59.466 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 136 16 3599 3583 1.64639 0 - 0.36018 2024-04-22T06:18:00.466 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 137 16 3599 3583 1.63437 0 - 0.36018 2024-04-22T06:18:01.466 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 138 16 3599 3583 1.62253 0 - 0.36018 2024-04-22T06:18:02.466 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 139 16 3599 3583 1.61085 0 - 0.36018 2024-04-22T06:18:02.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~52s 2024-04-22T06:18:02.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~52s 2024-04-22T06:18:02.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~52s 2024-04-22T06:18:02.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~52s 2024-04-22T06:18:03.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:18:03.467841+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:18:03.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:18:03.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 140 16 3599 3583 1.59935 0 - 0.36018 2024-04-22T06:18:04.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 141 16 3599 3583 1.588 0 - 0.36018 2024-04-22T06:18:05.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 142 16 3599 3583 1.57682 0 - 0.36018 2024-04-22T06:18:06.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 143 16 3599 3583 1.56579 0 - 0.36018 2024-04-22T06:18:07.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 144 16 3599 3583 1.55492 0 - 0.36018 2024-04-22T06:18:08.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~57s 2024-04-22T06:18:08.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~57s 2024-04-22T06:18:08.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~57s 2024-04-22T06:18:08.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~57s 2024-04-22T06:18:08.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 145 16 3599 3583 1.5442 0 - 0.36018 2024-04-22T06:18:09.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 146 16 3599 3583 1.53362 0 - 0.36018 2024-04-22T06:18:10.467 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 147 16 3599 3583 1.52319 0 - 0.36018 2024-04-22T06:18:11.468 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 148 16 3599 3583 1.51289 0 - 0.36018 2024-04-22T06:18:12.468 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 149 16 3599 3583 1.50274 0 - 0.36018 2024-04-22T06:18:13.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~62s 2024-04-22T06:18:13.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~62s 2024-04-22T06:18:13.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~62s 2024-04-22T06:18:13.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~62s 2024-04-22T06:18:13.468 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 150 16 3599 3583 1.49272 0 - 0.36018 2024-04-22T06:18:14.468 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 151 16 3599 3583 1.48284 0 - 0.36018 2024-04-22T06:18:15.468 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 152 16 3599 3583 1.47308 0 - 0.36018 2024-04-22T06:18:16.468 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 153 16 3599 3583 1.46345 0 - 0.36018 2024-04-22T06:18:17.469 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 154 16 3599 3583 1.45395 0 - 0.36018 2024-04-22T06:18:18.469 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 155 16 3599 3583 1.44457 0 - 0.36018 2024-04-22T06:18:18.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~68s 2024-04-22T06:18:18.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~68s 2024-04-22T06:18:18.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~68s 2024-04-22T06:18:18.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~68s 2024-04-22T06:18:19.469 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 156 16 3599 3583 1.43531 0 - 0.36018 2024-04-22T06:18:20.469 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 157 16 3599 3583 1.42617 0 - 0.36018 2024-04-22T06:18:21.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 158 16 3599 3583 1.41714 0 - 0.36018 2024-04-22T06:18:22.469 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 159 16 3599 3583 1.40823 0 - 0.36018 2024-04-22T06:18:23.469 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:18:23.470704+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:18:23.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:18:23.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 160 16 3599 3583 1.39943 0 - 0.36018 2024-04-22T06:18:23.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~73s 2024-04-22T06:18:23.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~73s 2024-04-22T06:18:23.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~73s 2024-04-22T06:18:23.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~73s 2024-04-22T06:18:24.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 161 16 3599 3583 1.39073 0 - 0.36018 2024-04-22T06:18:25.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 162 16 3599 3583 1.38215 0 - 0.36018 2024-04-22T06:18:26.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 163 16 3599 3583 1.37367 0 - 0.36018 2024-04-22T06:18:27.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 164 16 3599 3583 1.36529 0 - 0.36018 2024-04-22T06:18:28.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 165 16 3599 3583 1.35702 0 - 0.36018 2024-04-22T06:18:28.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~78s 2024-04-22T06:18:28.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~78s 2024-04-22T06:18:28.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~78s 2024-04-22T06:18:28.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~78s 2024-04-22T06:18:29.470 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 166 16 3599 3583 1.34884 0 - 0.36018 2024-04-22T06:18:30.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 167 16 3599 3583 1.34077 0 - 0.36018 2024-04-22T06:18:31.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 168 16 3599 3583 1.33279 0 - 0.36018 2024-04-22T06:18:32.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 169 16 3599 3583 1.3249 0 - 0.36018 2024-04-22T06:18:33.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 170 16 3599 3583 1.3171 0 - 0.36018 2024-04-22T06:18:34.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~83s 2024-04-22T06:18:34.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s 2024-04-22T06:18:34.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~83s 2024-04-22T06:18:34.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~83s 2024-04-22T06:18:34.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 171 16 3599 3583 1.3094 0 - 0.36018 2024-04-22T06:18:35.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 172 16 3599 3583 1.30179 0 - 0.36018 2024-04-22T06:18:36.471 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 173 16 3599 3583 1.29426 0 - 0.36018 2024-04-22T06:18:37.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 174 16 3599 3583 1.28683 0 - 0.36018 2024-04-22T06:18:38.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 175 16 3599 3583 1.27947 0 - 0.36018 2024-04-22T06:18:39.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~88s 2024-04-22T06:18:39.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~88s 2024-04-22T06:18:39.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~88s 2024-04-22T06:18:39.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~88s 2024-04-22T06:18:39.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 176 16 3599 3583 1.2722 0 - 0.36018 2024-04-22T06:18:40.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 177 16 3599 3583 1.26502 0 - 0.36018 2024-04-22T06:18:41.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 178 16 3599 3583 1.25791 0 - 0.36018 2024-04-22T06:18:42.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 179 16 3599 3583 1.25088 0 - 0.36018 2024-04-22T06:18:43.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:18:43.473598+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:18:43.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:18:43.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 180 16 3599 3583 1.24393 0 - 0.36018 2024-04-22T06:18:44.472 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 181 16 3599 3583 1.23706 0 - 0.36018 2024-04-22T06:18:44.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~94s 2024-04-22T06:18:44.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~94s 2024-04-22T06:18:44.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~94s 2024-04-22T06:18:44.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~94s 2024-04-22T06:18:45.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 182 16 3599 3583 1.23026 0 - 0.36018 2024-04-22T06:18:46.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 183 16 3599 3583 1.22354 0 - 0.36018 2024-04-22T06:18:47.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 184 16 3599 3583 1.21689 0 - 0.36018 2024-04-22T06:18:48.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 185 16 3599 3583 1.21031 0 - 0.36018 2024-04-22T06:18:49.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 186 16 3599 3583 1.2038 0 - 0.36018 2024-04-22T06:18:49.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~99s 2024-04-22T06:18:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~99s 2024-04-22T06:18:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2024-04-22T06:18:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~99s 2024-04-22T06:18:50.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 187 16 3599 3583 1.19737 0 - 0.36018 2024-04-22T06:18:51.473 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 188 16 3599 3583 1.191 0 - 0.36018 2024-04-22T06:18:52.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 189 16 3599 3583 1.1847 0 - 0.36018 2024-04-22T06:18:53.474 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 190 16 3599 3583 1.17846 0 - 0.36018 2024-04-22T06:18:54.474 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 191 16 3599 3583 1.17229 0 - 0.36018 2024-04-22T06:18:54.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~104s 2024-04-22T06:18:54.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~104s 2024-04-22T06:18:54.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~104s 2024-04-22T06:18:54.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~104s 2024-04-22T06:18:55.474 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 192 16 3599 3583 1.16619 0 - 0.36018 2024-04-22T06:18:56.474 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 193 16 3599 3583 1.16014 0 - 0.36018 2024-04-22T06:18:57.474 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 194 16 3599 3583 1.15416 0 - 0.36018 2024-04-22T06:18:58.474 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 195 16 3599 3583 1.14824 0 - 0.36018 2024-04-22T06:18:59.478 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 196 16 3599 3583 1.14239 0 - 0.36018 2024-04-22T06:19:00.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~109s 2024-04-22T06:19:00.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~109s 2024-04-22T06:19:00.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~109s 2024-04-22T06:19:00.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~109s 2024-04-22T06:19:00.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 197 16 3599 3583 1.13659 0 - 0.36018 2024-04-22T06:19:01.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 198 16 3599 3583 1.13085 0 - 0.36018 2024-04-22T06:19:02.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 199 16 3599 3583 1.12516 0 - 0.36018 2024-04-22T06:19:03.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:19:03.476356+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:19:03.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:19:03.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 200 16 3599 3583 1.11954 0 - 0.36018 2024-04-22T06:19:04.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 201 16 3599 3583 1.11397 0 - 0.36018 2024-04-22T06:19:05.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~114s 2024-04-22T06:19:05.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~114s 2024-04-22T06:19:05.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~114s 2024-04-22T06:19:05.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~114s 2024-04-22T06:19:05.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 202 16 3599 3583 1.10845 0 - 0.36018 2024-04-22T06:19:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:19:05.820 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:19:06.475 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 203 16 3599 3583 1.10299 0 - 0.36018 2024-04-22T06:19:07.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 204 16 3599 3583 1.09759 0 - 0.36018 2024-04-22T06:19:07.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:19:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:19:07.942+0000 7f0744f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:07.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:19:07.949+0000 7f0744f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:08.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 205 16 3599 3583 1.09223 0 - 0.36018 2024-04-22T06:19:09.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 206 16 3599 3583 1.08693 0 - 0.36018 2024-04-22T06:19:10.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 207 16 3599 3583 1.08168 0 - 0.36018 2024-04-22T06:19:10.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~120s 2024-04-22T06:19:10.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2024-04-22T06:19:10.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~120s 2024-04-22T06:19:10.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2024-04-22T06:19:11.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 208 16 3599 3583 1.07648 0 - 0.36018 2024-04-22T06:19:12.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 209 16 3599 3583 1.07133 0 - 0.36018 2024-04-22T06:19:13.476 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 210 16 3599 3583 1.06623 0 - 0.36018 2024-04-22T06:19:14.477 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 211 16 3599 3583 1.06117 0 - 0.36018 2024-04-22T06:19:15.477 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 212 16 3599 3583 1.05617 0 - 0.36018 2024-04-22T06:19:15.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~125s 2024-04-22T06:19:15.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2024-04-22T06:19:15.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~125s 2024-04-22T06:19:15.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~125s 2024-04-22T06:19:16.477 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 213 16 3599 3583 1.05121 0 - 0.36018 2024-04-22T06:19:17.477 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 214 16 3599 3583 1.0463 0 - 0.36018 2024-04-22T06:19:18.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 215 16 3599 3583 1.04143 0 - 0.36018 2024-04-22T06:19:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:19:19.402 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T06:19:19.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T06:19:19.477 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 216 16 3599 3583 1.03661 0 - 0.36018 2024-04-22T06:19:19.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:19:19.523+0000 7f622fd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:19:19.532+0000 7f622fd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:20.477 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 217 16 3599 3583 1.03183 0 - 0.36018 2024-04-22T06:19:20.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2024-04-22T06:19:20.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-04-22T06:19:20.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2024-04-22T06:19:20.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2024-04-22T06:19:21.478 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 218 16 3599 3583 1.0271 0 - 0.36018 2024-04-22T06:19:22.478 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 219 16 3599 3583 1.02241 0 - 0.36018 2024-04-22T06:19:23.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:19:23.479267+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:19:23.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:19:23.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 220 16 3599 3583 1.01776 0 - 0.36018 2024-04-22T06:19:24.478 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 221 16 3599 3583 1.01316 0 - 0.36018 2024-04-22T06:19:25.478 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 222 16 3599 3583 1.00859 0 - 0.36018 2024-04-22T06:19:26.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2024-04-22T06:19:26.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-04-22T06:19:26.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2024-04-22T06:19:26.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~135s 2024-04-22T06:19:26.478 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 223 16 3599 3583 1.00407 0 - 0.36018 2024-04-22T06:19:27.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 224 16 3599 3583 0.999586 0 - 0.36018 2024-04-22T06:19:28.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 225 16 3599 3583 0.995143 0 - 0.36018 2024-04-22T06:19:29.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 226 16 3599 3583 0.99074 0 - 0.36018 2024-04-22T06:19:30.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 227 16 3599 3583 0.986376 0 - 0.36018 2024-04-22T06:19:31.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~140s 2024-04-22T06:19:31.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-04-22T06:19:31.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2024-04-22T06:19:31.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~140s 2024-04-22T06:19:31.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 228 16 3599 3583 0.982049 0 - 0.36018 2024-04-22T06:19:32.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 229 16 3599 3583 0.977761 0 - 0.36018 2024-04-22T06:19:33.479 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 230 16 3599 3583 0.97351 0 - 0.36018 2024-04-22T06:19:34.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 231 16 3599 3583 0.969295 0 - 0.36018 2024-04-22T06:19:35.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 232 16 3599 3583 0.965117 0 - 0.36018 2024-04-22T06:19:36.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 233 16 3599 3583 0.960975 0 - 0.36018 2024-04-22T06:19:36.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~146s 2024-04-22T06:19:36.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2024-04-22T06:19:36.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~146s 2024-04-22T06:19:36.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~146s 2024-04-22T06:19:37.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 234 16 3599 3583 0.956868 0 - 0.36018 2024-04-22T06:19:38.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 235 16 3599 3583 0.952797 0 - 0.36018 2024-04-22T06:19:39.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 236 16 3599 3583 0.948759 0 - 0.36018 2024-04-22T06:19:40.480 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 237 16 3599 3583 0.944756 0 - 0.36018 2024-04-22T06:19:41.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 238 16 3599 3583 0.940786 0 - 0.36018 2024-04-22T06:19:41.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~151s 2024-04-22T06:19:41.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2024-04-22T06:19:41.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~151s 2024-04-22T06:19:41.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~151s 2024-04-22T06:19:42.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 239 16 3599 3583 0.93685 0 - 0.36018 2024-04-22T06:19:43.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:19:43.482207+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:19:43.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:19:43.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 240 16 3599 3583 0.932947 0 - 0.36018 2024-04-22T06:19:44.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 241 16 3599 3583 0.929075 0 - 0.36018 2024-04-22T06:19:45.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 242 16 3599 3583 0.925236 0 - 0.36018 2024-04-22T06:19:46.481 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 243 16 3599 3583 0.921429 0 - 0.36018 2024-04-22T06:19:46.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~156s 2024-04-22T06:19:46.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~156s 2024-04-22T06:19:46.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~156s 2024-04-22T06:19:46.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~156s 2024-04-22T06:19:47.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 244 16 3599 3583 0.917652 0 - 0.36018 2024-04-22T06:19:48.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 245 16 3599 3583 0.913907 0 - 0.36018 2024-04-22T06:19:49.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 246 16 3599 3583 0.910192 0 - 0.36018 2024-04-22T06:19:50.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 247 16 3599 3583 0.906507 0 - 0.36018 2024-04-22T06:19:51.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 248 16 3599 3583 0.902851 0 - 0.36018 2024-04-22T06:19:52.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~161s 2024-04-22T06:19:52.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~161s 2024-04-22T06:19:52.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~161s 2024-04-22T06:19:52.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~161s 2024-04-22T06:19:52.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 249 16 3599 3583 0.899225 0 - 0.36018 2024-04-22T06:19:53.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 250 16 3599 3583 0.895629 0 - 0.36018 2024-04-22T06:19:54.482 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 251 16 3599 3583 0.89206 0 - 0.36018 2024-04-22T06:19:55.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 252 16 3599 3583 0.88852 0 - 0.36018 2024-04-22T06:19:56.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 253 16 3599 3583 0.885008 0 - 0.36018 2024-04-22T06:19:57.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~166s 2024-04-22T06:19:57.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~166s 2024-04-22T06:19:57.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~166s 2024-04-22T06:19:57.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~166s 2024-04-22T06:19:57.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 254 16 3599 3583 0.881524 0 - 0.36018 2024-04-22T06:19:58.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 255 16 3599 3583 0.878067 0 - 0.36018 2024-04-22T06:19:59.483 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 256 16 3599 3583 0.874637 0 - 0.36018 2024-04-22T06:20:00.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 257 16 3599 3583 0.871234 0 - 0.36018 2024-04-22T06:20:01.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 258 16 3599 3583 0.867857 0 - 0.36018 2024-04-22T06:20:02.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 259 16 3599 3583 0.864506 0 - 0.36018 2024-04-22T06:20:02.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~172s 2024-04-22T06:20:02.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~172s 2024-04-22T06:20:02.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~172s 2024-04-22T06:20:02.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~172s 2024-04-22T06:20:03.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:20:03.485112+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:20:03.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:20:03.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 260 16 3599 3583 0.861181 0 - 0.36018 2024-04-22T06:20:04.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 261 16 3599 3583 0.857881 0 - 0.36018 2024-04-22T06:20:05.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 262 16 3599 3583 0.854607 0 - 0.36018 2024-04-22T06:20:06.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 263 16 3599 3583 0.851357 0 - 0.36018 2024-04-22T06:20:07.484 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 264 16 3599 3583 0.848132 0 - 0.36018 2024-04-22T06:20:07.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~177s 2024-04-22T06:20:07.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~177s 2024-04-22T06:20:07.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~177s 2024-04-22T06:20:07.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~177s 2024-04-22T06:20:08.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 265 16 3599 3583 0.844932 0 - 0.36018 2024-04-22T06:20:09.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 266 16 3599 3583 0.841755 0 - 0.36018 2024-04-22T06:20:10.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 267 16 3599 3583 0.838603 0 - 0.36018 2024-04-22T06:20:11.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 268 16 3599 3583 0.835474 0 - 0.36018 2024-04-22T06:20:12.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 269 16 3599 3583 0.832368 0 - 0.36018 2024-04-22T06:20:12.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~182s 2024-04-22T06:20:12.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2024-04-22T06:20:12.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~182s 2024-04-22T06:20:12.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~182s 2024-04-22T06:20:13.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 270 16 3599 3583 0.829285 0 - 0.36018 2024-04-22T06:20:14.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 271 16 3599 3583 0.826225 0 - 0.36018 2024-04-22T06:20:15.485 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 272 16 3599 3583 0.823187 0 - 0.36018 2024-04-22T06:20:16.486 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 273 16 3599 3583 0.820172 0 - 0.36018 2024-04-22T06:20:17.486 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 274 16 3599 3583 0.817179 0 - 0.36018 2024-04-22T06:20:18.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~187s 2024-04-22T06:20:18.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2024-04-22T06:20:18.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~187s 2024-04-22T06:20:18.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~187s 2024-04-22T06:20:18.486 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 275 16 3599 3583 0.814207 0 - 0.36018 2024-04-22T06:20:19.486 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 276 16 3599 3583 0.811257 0 - 0.36018 2024-04-22T06:20:20.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 277 16 3599 3583 0.808328 0 - 0.36018 2024-04-22T06:20:21.486 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 278 16 3599 3583 0.805421 0 - 0.36018 2024-04-22T06:20:22.486 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 279 16 3599 3583 0.802534 0 - 0.36018 2024-04-22T06:20:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~193s 2024-04-22T06:20:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~193s 2024-04-22T06:20:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2024-04-22T06:20:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~193s 2024-04-22T06:20:23.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:20:23.487883+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:20:23.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:20:23.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 280 16 3599 3583 0.799668 0 - 0.36018 2024-04-22T06:20:24.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 281 16 3599 3583 0.796822 0 - 0.36018 2024-04-22T06:20:25.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 282 16 3599 3583 0.793996 0 - 0.36018 2024-04-22T06:20:26.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 283 16 3599 3583 0.791191 0 - 0.36018 2024-04-22T06:20:27.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 284 16 3599 3583 0.788405 0 - 0.36018 2024-04-22T06:20:28.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 285 16 3599 3583 0.785638 0 - 0.36018 2024-04-22T06:20:28.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~198s 2024-04-22T06:20:28.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2024-04-22T06:20:28.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~198s 2024-04-22T06:20:28.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~198s 2024-04-22T06:20:29.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 286 16 3599 3583 0.782891 0 - 0.36018 2024-04-22T06:20:30.487 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 287 16 3599 3583 0.780164 0 - 0.36018 2024-04-22T06:20:31.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 288 16 3599 3583 0.777455 0 - 0.36018 2024-04-22T06:20:32.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 289 16 3599 3583 0.774764 0 - 0.36018 2024-04-22T06:20:33.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 290 16 3599 3583 0.772093 0 - 0.36018 2024-04-22T06:20:33.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~203s 2024-04-22T06:20:33.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~203s 2024-04-22T06:20:33.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~203s 2024-04-22T06:20:33.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~203s 2024-04-22T06:20:34.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 291 16 3599 3583 0.76944 0 - 0.36018 2024-04-22T06:20:35.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 292 16 3599 3583 0.766804 0 - 0.36018 2024-04-22T06:20:36.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 293 16 3599 3583 0.764187 0 - 0.36018 2024-04-22T06:20:37.488 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 294 16 3599 3583 0.761588 0 - 0.36018 2024-04-22T06:20:38.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 295 16 3599 3583 0.759006 0 - 0.36018 2024-04-22T06:20:38.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~208s 2024-04-22T06:20:38.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~208s 2024-04-22T06:20:38.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~208s 2024-04-22T06:20:38.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~208s 2024-04-22T06:20:39.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 296 16 3599 3583 0.756442 0 - 0.36018 2024-04-22T06:20:40.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 297 16 3599 3583 0.753895 0 - 0.36018 2024-04-22T06:20:41.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 298 16 3599 3583 0.751365 0 - 0.36018 2024-04-22T06:20:42.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 299 16 3599 3583 0.748852 0 - 0.36018 2024-04-22T06:20:43.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:20:43.490792+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:20:43.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:20:43.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 300 16 3599 3583 0.746356 0 - 0.36018 2024-04-22T06:20:44.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~213s 2024-04-22T06:20:44.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~213s 2024-04-22T06:20:44.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~213s 2024-04-22T06:20:44.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~213s 2024-04-22T06:20:44.489 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 301 16 3599 3583 0.743876 0 - 0.36018 2024-04-22T06:20:45.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 302 16 3599 3583 0.741413 0 - 0.36018 2024-04-22T06:20:46.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 303 16 3599 3583 0.738966 0 - 0.36018 2024-04-22T06:20:47.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 304 16 3599 3583 0.736535 0 - 0.36018 2024-04-22T06:20:48.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 305 16 3599 3583 0.734121 0 - 0.36018 2024-04-22T06:20:49.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~219s 2024-04-22T06:20:49.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~219s 2024-04-22T06:20:49.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~219s 2024-04-22T06:20:49.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~219s 2024-04-22T06:20:49.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 306 16 3599 3583 0.731721 0 - 0.36018 2024-04-22T06:20:50.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 307 16 3599 3583 0.729338 0 - 0.36018 2024-04-22T06:20:51.490 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 308 16 3599 3583 0.72697 0 - 0.36018 2024-04-22T06:20:52.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 309 16 3599 3583 0.724617 0 - 0.36018 2024-04-22T06:20:53.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 310 16 3599 3583 0.72228 0 - 0.36018 2024-04-22T06:20:54.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 311 16 3599 3583 0.719957 0 - 0.36018 2024-04-22T06:20:54.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~224s 2024-04-22T06:20:54.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~224s 2024-04-22T06:20:54.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~224s 2024-04-22T06:20:54.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~224s 2024-04-22T06:20:55.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 312 16 3599 3583 0.71765 0 - 0.36018 2024-04-22T06:20:56.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 313 16 3599 3583 0.715357 0 - 0.36018 2024-04-22T06:20:57.491 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 314 16 3599 3583 0.713079 0 - 0.36018 2024-04-22T06:20:58.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 315 16 3599 3583 0.710815 0 - 0.36018 2024-04-22T06:20:59.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 316 16 3599 3583 0.708565 0 - 0.36018 2024-04-22T06:20:59.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~229s 2024-04-22T06:20:59.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~229s 2024-04-22T06:20:59.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~229s 2024-04-22T06:20:59.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~229s 2024-04-22T06:21:00.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 317 16 3599 3583 0.70633 0 - 0.36018 2024-04-22T06:21:01.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 318 16 3599 3583 0.704109 0 - 0.36018 2024-04-22T06:21:02.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 319 16 3599 3583 0.701902 0 - 0.36018 2024-04-22T06:21:03.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:21:03.493804+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:21:03.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:21:03.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 320 16 3599 3583 0.699708 0 - 0.36018 2024-04-22T06:21:04.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 321 16 3599 3583 0.697528 0 - 0.36018 2024-04-22T06:21:04.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~234s 2024-04-22T06:21:04.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~234s 2024-04-22T06:21:04.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~234s 2024-04-22T06:21:04.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~234s 2024-04-22T06:21:05.492 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 322 16 3599 3583 0.695362 0 - 0.36018 2024-04-22T06:21:06.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 323 16 3599 3583 0.693209 0 - 0.36018 2024-04-22T06:21:07.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 324 16 3599 3583 0.69107 0 - 0.36018 2024-04-22T06:21:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:21:07.857 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:21:08.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 325 16 3599 3583 0.688943 0 - 0.36018 2024-04-22T06:21:09.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 326 16 3599 3583 0.68683 0 - 0.36018 2024-04-22T06:21:09.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:21:09.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:21:09.982+0000 7f8fadd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:21:09.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:21:09.988+0000 7f8fadd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:21:10.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2024-04-22T06:21:10.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~239s 2024-04-22T06:21:10.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~239s 2024-04-22T06:21:10.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~239s 2024-04-22T06:21:10.493 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 327 16 3599 3583 0.68473 0 - 0.36018 2024-04-22T06:21:11.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 328 16 3599 3583 0.682642 0 - 0.36018 2024-04-22T06:21:12.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 329 16 3599 3583 0.680567 0 - 0.36018 2024-04-22T06:21:13.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 330 16 3599 3583 0.678505 0 - 0.36018 2024-04-22T06:21:14.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 331 16 3599 3583 0.676455 0 - 0.36018 2024-04-22T06:21:15.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~245s 2024-04-22T06:21:15.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~245s 2024-04-22T06:21:15.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~245s 2024-04-22T06:21:15.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~245s 2024-04-22T06:21:15.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 332 16 3599 3583 0.674417 0 - 0.36018 2024-04-22T06:21:16.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 333 16 3599 3583 0.672392 0 - 0.36018 2024-04-22T06:21:17.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 334 16 3599 3583 0.670379 0 - 0.36018 2024-04-22T06:21:18.494 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 335 16 3599 3583 0.668378 0 - 0.36018 2024-04-22T06:21:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:21:19.438 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T06:21:19.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T06:21:19.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 336 16 3599 3583 0.666389 0 - 0.36018 2024-04-22T06:21:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:21:19.559+0000 7f033295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:21:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:21:19.569+0000 7f033295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:21:20.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 337 16 3599 3583 0.664411 0 - 0.36018 2024-04-22T06:21:20.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2024-04-22T06:21:20.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~250s 2024-04-22T06:21:20.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~250s 2024-04-22T06:21:20.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~250s 2024-04-22T06:21:21.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 338 16 3599 3583 0.662445 0 - 0.36018 2024-04-22T06:21:22.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 339 16 3599 3583 0.660491 0 - 0.36018 2024-04-22T06:21:23.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:21:23.496714+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:21:23.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:21:23.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 340 16 3599 3583 0.658549 0 - 0.36018 2024-04-22T06:21:24.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 341 16 3599 3583 0.656617 0 - 0.36018 2024-04-22T06:21:25.495 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 342 16 3599 3583 0.654697 0 - 0.36018 2024-04-22T06:21:25.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2024-04-22T06:21:25.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2024-04-22T06:21:25.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2024-04-22T06:21:25.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2024-04-22T06:21:26.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 343 16 3599 3583 0.652789 0 - 0.36018 2024-04-22T06:21:27.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 344 16 3599 3583 0.650891 0 - 0.36018 2024-04-22T06:21:28.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 345 16 3599 3583 0.649004 0 - 0.36018 2024-04-22T06:21:29.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 346 16 3599 3583 0.647129 0 - 0.36018 2024-04-22T06:21:30.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 347 16 3599 3583 0.645264 0 - 0.36018 2024-04-22T06:21:30.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2024-04-22T06:21:30.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2024-04-22T06:21:30.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~260s 2024-04-22T06:21:30.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~260s 2024-04-22T06:21:31.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 348 16 3599 3583 0.64341 0 - 0.36018 2024-04-22T06:21:32.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 349 16 3599 3583 0.641566 0 - 0.36018 2024-04-22T06:21:33.496 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 350 16 3599 3583 0.639733 0 - 0.36018 2024-04-22T06:21:34.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 351 16 3599 3583 0.63791 0 - 0.36018 2024-04-22T06:21:35.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 352 16 3599 3583 0.636098 0 - 0.36018 2024-04-22T06:21:36.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2024-04-22T06:21:36.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-04-22T06:21:36.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2024-04-22T06:21:36.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~265s 2024-04-22T06:21:36.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 353 16 3599 3583 0.634296 0 - 0.36018 2024-04-22T06:21:37.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 354 16 3599 3583 0.632504 0 - 0.36018 2024-04-22T06:21:38.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 355 16 3599 3583 0.630723 0 - 0.36018 2024-04-22T06:21:39.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 356 16 3599 3583 0.628951 0 - 0.36018 2024-04-22T06:21:40.497 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 357 16 3599 3583 0.627189 0 - 0.36018 2024-04-22T06:21:41.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~271s 2024-04-22T06:21:41.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~271s 2024-04-22T06:21:41.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~271s 2024-04-22T06:21:41.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~271s 2024-04-22T06:21:41.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 358 16 3599 3583 0.625437 0 - 0.36018 2024-04-22T06:21:42.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 359 16 3599 3583 0.623695 0 - 0.36018 2024-04-22T06:21:43.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:21:43.499405+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:21:43.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:21:43.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 360 16 3599 3583 0.621963 0 - 0.36018 2024-04-22T06:21:44.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 361 16 3599 3583 0.62024 0 - 0.36018 2024-04-22T06:21:45.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 362 16 3599 3583 0.618526 0 - 0.36018 2024-04-22T06:21:46.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 363 16 3599 3583 0.616822 0 - 0.36018 2024-04-22T06:21:46.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2024-04-22T06:21:46.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-04-22T06:21:46.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2024-04-22T06:21:46.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~276s 2024-04-22T06:21:47.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 364 16 3599 3583 0.615128 0 - 0.36018 2024-04-22T06:21:48.498 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 365 16 3599 3583 0.613443 0 - 0.36018 2024-04-22T06:21:49.499 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 366 16 3599 3583 0.611767 0 - 0.36018 2024-04-22T06:21:50.499 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 367 16 3599 3583 0.6101 0 - 0.36018 2024-04-22T06:21:51.499 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 368 16 3599 3583 0.608442 0 - 0.36018 2024-04-22T06:21:51.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s 2024-04-22T06:21:51.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2024-04-22T06:21:51.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2024-04-22T06:21:51.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~281s 2024-04-22T06:21:52.499 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 369 16 3599 3583 0.606793 0 - 0.36018 2024-04-22T06:21:53.499 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 370 16 3599 3583 0.605153 0 - 0.36018 2024-04-22T06:21:54.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 371 16 3599 3583 0.603522 0 - 0.36018 2024-04-22T06:21:55.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 372 16 3599 3583 0.601899 0 - 0.36018 2024-04-22T06:21:56.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 373 16 3599 3583 0.600286 0 - 0.36018 2024-04-22T06:21:56.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~286s 2024-04-22T06:21:56.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2024-04-22T06:21:56.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2024-04-22T06:21:56.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~286s 2024-04-22T06:21:57.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 374 16 3599 3583 0.598681 0 - 0.36018 2024-04-22T06:21:58.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 375 16 3599 3583 0.597084 0 - 0.36018 2024-04-22T06:21:59.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 376 16 3599 3583 0.595496 0 - 0.36018 2024-04-22T06:22:00.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 377 16 3599 3583 0.593917 0 - 0.36018 2024-04-22T06:22:01.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 378 16 3599 3583 0.592345 0 - 0.36018 2024-04-22T06:22:02.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~291s 2024-04-22T06:22:02.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~291s 2024-04-22T06:22:02.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~291s 2024-04-22T06:22:02.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~291s 2024-04-22T06:22:02.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 379 16 3599 3583 0.590782 0 - 0.36018 2024-04-22T06:22:03.500 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:22:03.502203+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:22:03.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:22:03.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 380 16 3599 3583 0.589228 0 - 0.36018 2024-04-22T06:22:04.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 381 16 3599 3583 0.587681 0 - 0.36018 2024-04-22T06:22:05.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 382 16 3599 3583 0.586143 0 - 0.36018 2024-04-22T06:22:06.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 383 16 3599 3583 0.584612 0 - 0.36018 2024-04-22T06:22:07.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~297s 2024-04-22T06:22:07.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2024-04-22T06:22:07.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2024-04-22T06:22:07.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~297s 2024-04-22T06:22:07.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 384 16 3599 3583 0.58309 0 - 0.36018 2024-04-22T06:22:08.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 385 16 3599 3583 0.581575 0 - 0.36018 2024-04-22T06:22:09.501 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 386 16 3599 3583 0.580069 0 - 0.36018 2024-04-22T06:22:10.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 387 16 3599 3583 0.57857 0 - 0.36018 2024-04-22T06:22:11.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 388 16 3599 3583 0.577079 0 - 0.36018 2024-04-22T06:22:12.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 389 16 3599 3583 0.575595 0 - 0.36018 2024-04-22T06:22:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~302s 2024-04-22T06:22:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~302s 2024-04-22T06:22:12.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~302s 2024-04-22T06:22:12.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~302s 2024-04-22T06:22:12.607 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-22T06:22:12.809 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-22T06:22:12.810 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-22T06:22:12.811 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-22T06:22:12.811+0000 7f4511dff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 27079) UID: 0 2024-04-22T06:22:12.811 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-22T06:22:12.811+0000 7f4511dff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T06:22:12.811 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:22:12.811+0000 7fd5d89f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 26914) UID: 0 2024-04-22T06:22:12.812 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-22T06:22:12.811+0000 7fd5d89f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-22T06:22:13.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 390 16 3599 3583 0.574119 0 - 0.36018 2024-04-22T06:22:14.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 391 16 3599 3583 0.572651 0 - 0.36018 2024-04-22T06:22:15.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 392 16 3599 3583 0.57119 0 - 0.36018 2024-04-22T06:22:16.502 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 393 16 3599 3583 0.569737 0 - 0.36018 2024-04-22T06:22:17.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 394 16 3599 3583 0.568291 0 - 0.36018 2024-04-22T06:22:18.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 395 16 3599 3583 0.566852 0 - 0.36018 2024-04-22T06:22:19.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 396 16 3599 3583 0.56542 0 - 0.36018 2024-04-22T06:22:20.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 397 16 3599 3583 0.563996 0 - 0.36018 2024-04-22T06:22:21.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 398 16 3599 3583 0.562579 0 - 0.36018 2024-04-22T06:22:22.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 399 16 3599 3583 0.561169 0 - 0.36018 2024-04-22T06:22:23.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:22:23.505083+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:22:23.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:22:23.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 400 16 3599 3583 0.559766 0 - 0.36018 2024-04-22T06:22:24.503 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 401 16 3599 3583 0.55837 0 - 0.36018 2024-04-22T06:22:25.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 402 16 3599 3583 0.556981 0 - 0.36018 2024-04-22T06:22:26.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 403 16 3599 3583 0.555599 0 - 0.36018 2024-04-22T06:22:27.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 404 16 3599 3583 0.554224 0 - 0.36018 2024-04-22T06:22:28.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 405 16 3599 3583 0.552855 0 - 0.36018 2024-04-22T06:22:29.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 406 16 3599 3583 0.551494 0 - 0.36018 2024-04-22T06:22:30.504 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 407 16 3599 3583 0.550139 0 - 0.36018 2024-04-22T06:22:31.505 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 408 16 3599 3583 0.54879 0 - 0.36018 2024-04-22T06:22:32.505 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 409 16 3599 3583 0.547448 0 - 0.36018 2024-04-22T06:22:33.505 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 410 16 3599 3583 0.546113 0 - 0.36018 2024-04-22T06:22:34.505 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 411 16 3599 3583 0.544785 0 - 0.36018 2024-04-22T06:22:35.505 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 412 16 3599 3583 0.543462 0 - 0.36018 2024-04-22T06:22:36.505 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 413 16 3599 3583 0.542146 0 - 0.36018 2024-04-22T06:22:37.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 414 16 3599 3583 0.540837 0 - 0.36018 2024-04-22T06:22:38.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 415 16 3599 3583 0.539534 0 - 0.36018 2024-04-22T06:22:39.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 416 16 3599 3583 0.538237 0 - 0.36018 2024-04-22T06:22:40.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 417 16 3599 3583 0.536946 0 - 0.36018 2024-04-22T06:22:41.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 418 16 3599 3583 0.535661 0 - 0.36018 2024-04-22T06:22:42.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 419 16 3599 3583 0.534383 0 - 0.36018 2024-04-22T06:22:43.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:22:43.508084+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:22:43.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:22:43.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 420 16 3599 3583 0.53311 0 - 0.36018 2024-04-22T06:22:44.506 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 421 16 3599 3583 0.531844 0 - 0.36018 2024-04-22T06:22:45.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 422 16 3599 3583 0.530584 0 - 0.36018 2024-04-22T06:22:46.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 423 16 3599 3583 0.529329 0 - 0.36018 2024-04-22T06:22:47.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 424 16 3599 3583 0.528081 0 - 0.36018 2024-04-22T06:22:48.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 425 16 3599 3583 0.526838 0 - 0.36018 2024-04-22T06:22:49.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 426 16 3599 3583 0.525602 0 - 0.36018 2024-04-22T06:22:50.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 427 16 3599 3583 0.524371 0 - 0.36018 2024-04-22T06:22:51.507 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 428 16 3599 3583 0.523146 0 - 0.36018 2024-04-22T06:22:52.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 429 16 3599 3583 0.521926 0 - 0.36018 2024-04-22T06:22:53.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 430 16 3599 3583 0.520712 0 - 0.36018 2024-04-22T06:22:54.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 431 16 3599 3583 0.519504 0 - 0.36018 2024-04-22T06:22:55.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 432 16 3599 3583 0.518302 0 - 0.36018 2024-04-22T06:22:56.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 433 16 3599 3583 0.517105 0 - 0.36018 2024-04-22T06:22:57.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 434 16 3599 3583 0.515913 0 - 0.36018 2024-04-22T06:22:58.508 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 435 16 3599 3583 0.514727 0 - 0.36018 2024-04-22T06:22:59.509 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 436 16 3599 3583 0.513547 0 - 0.36018 2024-04-22T06:23:00.509 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 437 16 3599 3583 0.512371 0 - 0.36018 2024-04-22T06:23:01.509 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 438 16 3599 3583 0.511202 0 - 0.36018 2024-04-22T06:23:02.509 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 439 16 3599 3583 0.510037 0 - 0.36018 2024-04-22T06:23:03.509 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:23:03.511072+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:23:03.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:23:03.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 440 16 3599 3583 0.508878 0 - 0.36018 2024-04-22T06:23:04.509 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 441 16 3599 3583 0.507724 0 - 0.36018 2024-04-22T06:23:05.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 442 16 3599 3583 0.506575 0 - 0.36018 2024-04-22T06:23:06.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 443 16 3599 3583 0.505432 0 - 0.36018 2024-04-22T06:23:07.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 444 16 3599 3583 0.504293 0 - 0.36018 2024-04-22T06:23:08.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 445 16 3599 3583 0.50316 0 - 0.36018 2024-04-22T06:23:09.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 446 16 3599 3583 0.502032 0 - 0.36018 2024-04-22T06:23:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:23:09.896 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:23:10.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 447 16 3599 3583 0.500909 0 - 0.36018 2024-04-22T06:23:11.510 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 448 16 3599 3583 0.499791 0 - 0.36018 2024-04-22T06:23:11.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:23:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:23:12.020+0000 7f700d020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:23:12.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 449 16 3599 3583 0.498678 0 - 0.36018 2024-04-22T06:23:13.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 450 16 3599 3583 0.497569 0 - 0.36018 2024-04-22T06:23:14.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 451 16 3599 3583 0.496466 0 - 0.36018 2024-04-22T06:23:15.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 452 16 3599 3583 0.495368 0 - 0.36018 2024-04-22T06:23:16.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 453 16 3599 3583 0.494274 0 - 0.36018 2024-04-22T06:23:17.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 454 16 3599 3583 0.493185 0 - 0.36018 2024-04-22T06:23:18.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 455 16 3599 3583 0.492102 0 - 0.36018 2024-04-22T06:23:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:23:19.473 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T06:23:19.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T06:23:19.511 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 456 16 3599 3583 0.491022 0 - 0.36018 2024-04-22T06:23:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:23:19.594+0000 7fd29106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:23:20.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 457 16 3599 3583 0.489948 0 - 0.36018 2024-04-22T06:23:21.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 458 16 3599 3583 0.488878 0 - 0.36018 2024-04-22T06:23:22.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 459 16 3599 3583 0.487813 0 - 0.36018 2024-04-22T06:23:23.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:23:23.514047+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:23:23.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:23:23.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 460 16 3599 3583 0.486753 0 - 0.36018 2024-04-22T06:23:24.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 461 16 3599 3583 0.485697 0 - 0.36018 2024-04-22T06:23:25.512 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 462 16 3599 3583 0.484645 0 - 0.36018 2024-04-22T06:23:26.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 463 16 3599 3583 0.483599 0 - 0.36018 2024-04-22T06:23:27.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 464 16 3599 3583 0.482556 0 - 0.36018 2024-04-22T06:23:28.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 465 16 3599 3583 0.481519 0 - 0.36018 2024-04-22T06:23:29.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 466 16 3599 3583 0.480485 0 - 0.36018 2024-04-22T06:23:30.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 467 16 3599 3583 0.479456 0 - 0.36018 2024-04-22T06:23:31.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 468 16 3599 3583 0.478432 0 - 0.36018 2024-04-22T06:23:32.513 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 469 16 3599 3583 0.477412 0 - 0.36018 2024-04-22T06:23:33.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 470 16 3599 3583 0.476396 0 - 0.36018 2024-04-22T06:23:34.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 471 16 3599 3583 0.475385 0 - 0.36018 2024-04-22T06:23:35.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 472 16 3599 3583 0.474377 0 - 0.36018 2024-04-22T06:23:36.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 473 16 3599 3583 0.473375 0 - 0.36018 2024-04-22T06:23:37.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 474 16 3599 3583 0.472376 0 - 0.36018 2024-04-22T06:23:38.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 475 16 3599 3583 0.471381 0 - 0.36018 2024-04-22T06:23:39.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 476 16 3599 3583 0.470391 0 - 0.36018 2024-04-22T06:23:40.514 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 477 16 3599 3583 0.469405 0 - 0.36018 2024-04-22T06:23:41.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 478 16 3599 3583 0.468423 0 - 0.36018 2024-04-22T06:23:42.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 479 16 3599 3583 0.467445 0 - 0.36018 2024-04-22T06:23:43.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:23:43.516913+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:23:43.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:23:43.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 480 16 3599 3583 0.466471 0 - 0.36018 2024-04-22T06:23:44.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 481 16 3599 3583 0.465501 0 - 0.36018 2024-04-22T06:23:45.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 482 16 3599 3583 0.464536 0 - 0.36018 2024-04-22T06:23:46.515 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 483 16 3599 3583 0.463574 0 - 0.36018 2024-04-22T06:23:47.516 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 484 16 3599 3583 0.462616 0 - 0.36018 2024-04-22T06:23:48.516 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 485 16 3599 3583 0.461662 0 - 0.36018 2024-04-22T06:23:49.516 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 486 16 3599 3583 0.460712 0 - 0.36018 2024-04-22T06:23:50.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 487 16 3599 3583 0.459766 0 - 0.36018 2024-04-22T06:23:51.516 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 488 16 3599 3583 0.458824 0 - 0.36018 2024-04-22T06:23:52.516 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 489 16 3599 3583 0.457886 0 - 0.36018 2024-04-22T06:23:53.516 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 490 16 3599 3583 0.456951 0 - 0.36018 2024-04-22T06:23:54.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 491 16 3599 3583 0.456021 0 - 0.36018 2024-04-22T06:23:55.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 492 16 3599 3583 0.455094 0 - 0.36018 2024-04-22T06:23:56.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 493 16 3599 3583 0.454171 0 - 0.36018 2024-04-22T06:23:57.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 494 16 3599 3583 0.453251 0 - 0.36018 2024-04-22T06:23:58.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 495 16 3599 3583 0.452336 0 - 0.36018 2024-04-22T06:23:59.517 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 496 16 3599 3583 0.451424 0 - 0.36018 2024-04-22T06:24:00.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 497 16 3599 3583 0.450515 0 - 0.36018 2024-04-22T06:24:01.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 498 16 3599 3583 0.449611 0 - 0.36018 2024-04-22T06:24:02.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 499 16 3599 3583 0.44871 0 - 0.36018 2024-04-22T06:24:03.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:24:03.519830+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:24:03.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:24:03.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 500 16 3599 3583 0.447812 0 - 0.36018 2024-04-22T06:24:04.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 501 16 3599 3583 0.446918 0 - 0.36018 2024-04-22T06:24:05.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 502 16 3599 3583 0.446028 0 - 0.36018 2024-04-22T06:24:06.518 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 503 16 3599 3583 0.445141 0 - 0.36018 2024-04-22T06:24:07.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 504 16 3599 3583 0.444258 0 - 0.36018 2024-04-22T06:24:08.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 505 16 3599 3583 0.443378 0 - 0.36018 2024-04-22T06:24:09.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 506 16 3599 3583 0.442502 0 - 0.36018 2024-04-22T06:24:10.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 507 16 3599 3583 0.441629 0 - 0.36018 2024-04-22T06:24:11.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 508 16 3599 3583 0.44076 0 - 0.36018 2024-04-22T06:24:12.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 509 16 3599 3583 0.439894 0 - 0.36018 2024-04-22T06:24:13.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 510 16 3599 3583 0.439032 0 - 0.36018 2024-04-22T06:24:14.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 511 16 3599 3583 0.438172 0 - 0.36018 2024-04-22T06:24:15.519 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 512 16 3599 3583 0.437317 0 - 0.36018 2024-04-22T06:24:16.520 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 513 16 3599 3583 0.436464 0 - 0.36018 2024-04-22T06:24:17.520 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 514 16 3599 3583 0.435615 0 - 0.36018 2024-04-22T06:24:18.520 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 515 16 3599 3583 0.434769 0 - 0.36018 2024-04-22T06:24:19.520 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 516 16 3599 3583 0.433927 0 - 0.36018 2024-04-22T06:24:20.520 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 517 16 3599 3583 0.433087 0 - 0.36018 2024-04-22T06:24:21.520 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 518 16 3599 3583 0.432251 0 - 0.36018 2024-04-22T06:24:22.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 519 16 3599 3583 0.431418 0 - 0.36018 2024-04-22T06:24:23.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:24:23.522605+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:24:23.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:24:23.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 520 16 3599 3583 0.430589 0 - 0.36018 2024-04-22T06:24:24.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 521 16 3599 3583 0.429762 0 - 0.36018 2024-04-22T06:24:25.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 522 16 3599 3583 0.428939 0 - 0.36018 2024-04-22T06:24:26.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 523 16 3599 3583 0.428119 0 - 0.36018 2024-04-22T06:24:27.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 524 16 3599 3583 0.427302 0 - 0.36018 2024-04-22T06:24:28.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 525 16 3599 3583 0.426488 0 - 0.36018 2024-04-22T06:24:29.521 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 526 16 3599 3583 0.425677 0 - 0.36018 2024-04-22T06:24:30.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 527 16 3599 3583 0.424869 0 - 0.36018 2024-04-22T06:24:31.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 528 16 3599 3583 0.424065 0 - 0.36018 2024-04-22T06:24:32.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 529 16 3599 3583 0.423263 0 - 0.36018 2024-04-22T06:24:33.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 530 16 3599 3583 0.422464 0 - 0.36018 2024-04-22T06:24:34.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 531 16 3599 3583 0.421669 0 - 0.36018 2024-04-22T06:24:35.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 532 16 3599 3583 0.420876 0 - 0.36018 2024-04-22T06:24:36.522 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 533 16 3599 3583 0.420086 0 - 0.36018 2024-04-22T06:24:37.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 534 16 3599 3583 0.4193 0 - 0.36018 2024-04-22T06:24:38.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 535 16 3599 3583 0.418516 0 - 0.36018 2024-04-22T06:24:39.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 536 16 3599 3583 0.417735 0 - 0.36018 2024-04-22T06:24:40.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 537 16 3599 3583 0.416957 0 - 0.36018 2024-04-22T06:24:41.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 538 16 3599 3583 0.416182 0 - 0.36018 2024-04-22T06:24:42.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 539 16 3599 3583 0.41541 0 - 0.36018 2024-04-22T06:24:43.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:24:43.525486+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:24:43.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:24:43.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 540 16 3599 3583 0.414641 0 - 0.36018 2024-04-22T06:24:44.523 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 541 16 3599 3583 0.413874 0 - 0.36018 2024-04-22T06:24:45.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 542 16 3599 3583 0.413111 0 - 0.36018 2024-04-22T06:24:46.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 543 16 3599 3583 0.41235 0 - 0.36018 2024-04-22T06:24:47.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 544 16 3599 3583 0.411592 0 - 0.36018 2024-04-22T06:24:48.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 545 16 3599 3583 0.410837 0 - 0.36018 2024-04-22T06:24:49.524 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 546 16 3599 3583 0.410084 0 - 0.36018 2024-04-22T06:24:50.525 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 547 16 3599 3583 0.409335 0 - 0.36018 2024-04-22T06:24:51.525 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 548 16 3599 3583 0.408588 0 - 0.36018 2024-04-22T06:24:52.525 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 549 16 3599 3583 0.407843 0 - 0.36018 2024-04-22T06:24:53.525 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 550 16 3599 3583 0.407102 0 - 0.36018 2024-04-22T06:24:54.525 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 551 16 3599 3583 0.406363 0 - 0.36018 2024-04-22T06:24:55.525 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 552 16 3599 3583 0.405627 0 - 0.36018 2024-04-22T06:24:56.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 553 16 3599 3583 0.404893 0 - 0.36018 2024-04-22T06:24:57.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 554 16 3599 3583 0.404163 0 - 0.36018 2024-04-22T06:24:58.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 555 16 3599 3583 0.403434 0 - 0.36018 2024-04-22T06:24:59.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 556 16 3599 3583 0.402709 0 - 0.36018 2024-04-22T06:25:00.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 557 16 3599 3583 0.401986 0 - 0.36018 2024-04-22T06:25:01.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 558 16 3599 3583 0.401265 0 - 0.36018 2024-04-22T06:25:02.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 559 16 3599 3583 0.400547 0 - 0.36018 2024-04-22T06:25:03.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:25:03.528323+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:25:03.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:25:03.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 560 16 3599 3583 0.399832 0 - 0.36018 2024-04-22T06:25:04.526 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 561 16 3599 3583 0.39912 0 - 0.36018 2024-04-22T06:25:05.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 562 16 3599 3583 0.398409 0 - 0.36018 2024-04-22T06:25:06.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 563 16 3599 3583 0.397702 0 - 0.36018 2024-04-22T06:25:07.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 564 16 3599 3583 0.396997 0 - 0.36018 2024-04-22T06:25:08.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 565 16 3599 3583 0.396294 0 - 0.36018 2024-04-22T06:25:09.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 566 16 3599 3583 0.395594 0 - 0.36018 2024-04-22T06:25:10.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 567 16 3599 3583 0.394896 0 - 0.36018 2024-04-22T06:25:11.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 568 16 3599 3583 0.394201 0 - 0.36018 2024-04-22T06:25:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:25:11.934 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:25:12.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 569 16 3599 3583 0.393508 0 - 0.36018 2024-04-22T06:25:13.527 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 570 16 3599 3583 0.392818 0 - 0.36018 2024-04-22T06:25:13.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:25:14.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:25:14.057+0000 7f6aaf66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:25:14.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 571 16 3599 3583 0.39213 0 - 0.36018 2024-04-22T06:25:15.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 572 16 3599 3583 0.391444 0 - 0.36018 2024-04-22T06:25:16.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 573 16 3599 3583 0.390761 0 - 0.36018 2024-04-22T06:25:17.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 574 16 3599 3583 0.39008 0 - 0.36018 2024-04-22T06:25:18.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 575 16 3599 3583 0.389402 0 - 0.36018 2024-04-22T06:25:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:25:19.509 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T06:25:19.528 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 576 16 3599 3583 0.388726 0 - 0.36018 2024-04-22T06:25:20.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 577 16 3599 3583 0.388052 0 - 0.36018 2024-04-22T06:25:21.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 578 16 3599 3583 0.387381 0 - 0.36018 2024-04-22T06:25:22.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 579 16 3599 3583 0.386712 0 - 0.36018 2024-04-22T06:25:23.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:25:23.531166+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:25:23.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:25:23.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 580 16 3599 3583 0.386045 0 - 0.36018 2024-04-22T06:25:24.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1] 2024-04-22T06:25:24.526 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T06:25:24.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:25:24.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 581 16 3599 3583 0.38538 0 - 0.36018 2024-04-22T06:25:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:25:24.650+0000 7faa6275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:25:25.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 582 16 3599 3583 0.384718 0 - 0.36018 2024-04-22T06:25:26.529 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 583 16 3599 3583 0.384058 0 - 0.36018 2024-04-22T06:25:27.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 584 16 3599 3583 0.383401 0 - 0.36018 2024-04-22T06:25:28.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 585 16 3599 3583 0.382745 0 - 0.36018 2024-04-22T06:25:29.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 586 16 3599 3583 0.382092 0 - 0.36018 2024-04-22T06:25:30.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 587 16 3599 3583 0.381441 0 - 0.36018 2024-04-22T06:25:31.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 588 16 3599 3583 0.380793 0 - 0.36018 2024-04-22T06:25:32.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 589 16 3599 3583 0.380146 0 - 0.36018 2024-04-22T06:25:33.530 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 590 16 3599 3583 0.379502 0 - 0.36018 2024-04-22T06:25:34.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 591 16 3599 3583 0.37886 0 - 0.36018 2024-04-22T06:25:35.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 592 16 3599 3583 0.37822 0 - 0.36018 2024-04-22T06:25:36.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 593 16 3599 3583 0.377582 0 - 0.36018 2024-04-22T06:25:37.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 594 16 3599 3583 0.376946 0 - 0.36018 2024-04-22T06:25:38.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 595 16 3599 3583 0.376313 0 - 0.36018 2024-04-22T06:25:39.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 596 16 3599 3583 0.375681 0 - 0.36018 2024-04-22T06:25:40.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 597 16 3599 3583 0.375052 0 - 0.36018 2024-04-22T06:25:41.531 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 598 16 3599 3583 0.374425 0 - 0.36018 2024-04-22T06:25:42.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 599 16 3599 3583 0.3738 0 - 0.36018 2024-04-22T06:25:43.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:25:43.533948+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:25:43.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:25:43.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 600 16 3599 3583 0.373177 0 - 0.36018 2024-04-22T06:25:44.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 601 16 3599 3583 0.372556 0 - 0.36018 2024-04-22T06:25:45.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 602 16 3599 3583 0.371937 0 - 0.36018 2024-04-22T06:25:46.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 603 16 3599 3583 0.37132 0 - 0.36018 2024-04-22T06:25:47.532 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 604 16 3599 3583 0.370705 0 - 0.36018 2024-04-22T06:25:48.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 605 16 3599 3583 0.370093 0 - 0.36018 2024-04-22T06:25:49.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 606 16 3599 3583 0.369482 0 - 0.36018 2024-04-22T06:25:50.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 607 16 3599 3583 0.368873 0 - 0.36018 2024-04-22T06:25:51.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 608 16 3599 3583 0.368266 0 - 0.36018 2024-04-22T06:25:52.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 609 16 3599 3583 0.367662 0 - 0.36018 2024-04-22T06:25:53.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 610 16 3599 3583 0.367059 0 - 0.36018 2024-04-22T06:25:54.533 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 611 16 3599 3583 0.366458 0 - 0.36018 2024-04-22T06:25:55.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 612 16 3599 3583 0.365859 0 - 0.36018 2024-04-22T06:25:56.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 613 16 3599 3583 0.365263 0 - 0.36018 2024-04-22T06:25:57.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 614 16 3599 3583 0.364668 0 - 0.36018 2024-04-22T06:25:58.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 615 16 3599 3583 0.364075 0 - 0.36018 2024-04-22T06:25:59.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 616 16 3599 3583 0.363484 0 - 0.36018 2024-04-22T06:26:00.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 617 16 3599 3583 0.362895 0 - 0.36018 2024-04-22T06:26:01.534 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 618 16 3599 3583 0.362307 0 - 0.36018 2024-04-22T06:26:02.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 619 16 3599 3583 0.361722 0 - 0.36018 2024-04-22T06:26:03.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:26:03.536919+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:26:03.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:26:03.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 620 16 3599 3583 0.361139 0 - 0.36018 2024-04-22T06:26:04.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 621 16 3599 3583 0.360557 0 - 0.36018 2024-04-22T06:26:05.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 622 16 3599 3583 0.359977 0 - 0.36018 2024-04-22T06:26:06.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 623 16 3599 3583 0.3594 0 - 0.36018 2024-04-22T06:26:07.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 624 16 3599 3583 0.358824 0 - 0.36018 2024-04-22T06:26:08.535 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 625 16 3599 3583 0.35825 0 - 0.36018 2024-04-22T06:26:09.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 626 16 3599 3583 0.357677 0 - 0.36018 2024-04-22T06:26:10.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 627 16 3599 3583 0.357107 0 - 0.36018 2024-04-22T06:26:11.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 628 16 3599 3583 0.356538 0 - 0.36018 2024-04-22T06:26:12.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 629 16 3599 3583 0.355971 0 - 0.36018 2024-04-22T06:26:13.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 630 16 3599 3583 0.355406 0 - 0.36018 2024-04-22T06:26:14.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 631 16 3599 3583 0.354843 0 - 0.36018 2024-04-22T06:26:15.536 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 632 16 3599 3583 0.354282 0 - 0.36018 2024-04-22T06:26:16.537 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 633 16 3599 3583 0.353722 0 - 0.36018 2024-04-22T06:26:17.537 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 634 16 3599 3583 0.353164 0 - 0.36018 2024-04-22T06:26:18.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 635 16 3599 3583 0.352608 0 - 0.36018 2024-04-22T06:26:19.537 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 636 16 3599 3583 0.352053 0 - 0.36018 2024-04-22T06:26:20.537 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 637 16 3599 3583 0.351501 0 - 0.36018 2024-04-22T06:26:21.537 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 638 16 3599 3583 0.35095 0 - 0.36018 2024-04-22T06:26:22.537 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 639 16 3599 3583 0.350401 0 - 0.36018 2024-04-22T06:26:23.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:26:23.539848+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:26:23.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:26:23.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 640 16 3599 3583 0.349853 0 - 0.36018 2024-04-22T06:26:24.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 641 16 3599 3583 0.349307 0 - 0.36018 2024-04-22T06:26:25.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 642 16 3599 3583 0.348763 0 - 0.36018 2024-04-22T06:26:26.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 643 16 3599 3583 0.348221 0 - 0.36018 2024-04-22T06:26:27.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 644 16 3599 3583 0.34768 0 - 0.36018 2024-04-22T06:26:28.538 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 645 16 3599 3583 0.347141 0 - 0.36018 2024-04-22T06:26:29.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 646 16 3599 3583 0.346604 0 - 0.36018 2024-04-22T06:26:30.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 647 16 3599 3583 0.346068 0 - 0.36018 2024-04-22T06:26:31.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 648 16 3599 3583 0.345534 0 - 0.36018 2024-04-22T06:26:32.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 649 16 3599 3583 0.345001 0 - 0.36018 2024-04-22T06:26:33.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 650 16 3599 3583 0.344471 0 - 0.36018 2024-04-22T06:26:34.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 651 16 3599 3583 0.343942 0 - 0.36018 2024-04-22T06:26:35.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 652 16 3599 3583 0.343414 0 - 0.36018 2024-04-22T06:26:36.539 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 653 16 3599 3583 0.342888 0 - 0.36018 2024-04-22T06:26:37.540 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 654 16 3599 3583 0.342364 0 - 0.36018 2024-04-22T06:26:38.540 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 655 16 3599 3583 0.341841 0 - 0.36018 2024-04-22T06:26:39.540 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 656 16 3599 3583 0.34132 0 - 0.36018 2024-04-22T06:26:40.540 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 657 16 3599 3583 0.3408 0 - 0.36018 2024-04-22T06:26:41.540 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 658 16 3599 3583 0.340283 0 - 0.36018 2024-04-22T06:26:42.540 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 659 16 3599 3583 0.339766 0 - 0.36018 2024-04-22T06:26:43.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:26:43.542789+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:26:43.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:26:43.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 660 16 3599 3583 0.339251 0 - 0.36018 2024-04-22T06:26:44.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 661 16 3599 3583 0.338738 0 - 0.36018 2024-04-22T06:26:45.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 662 16 3599 3583 0.338226 0 - 0.36018 2024-04-22T06:26:46.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 663 16 3599 3583 0.337716 0 - 0.36018 2024-04-22T06:26:47.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 664 16 3599 3583 0.337208 0 - 0.36018 2024-04-22T06:26:48.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 665 16 3599 3583 0.336701 0 - 0.36018 2024-04-22T06:26:49.542 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 666 16 3599 3583 0.336195 0 - 0.36018 2024-04-22T06:26:50.541 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 667 16 3599 3583 0.335691 0 - 0.36018 2024-04-22T06:26:51.542 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 668 16 3599 3583 0.335188 0 - 0.36018 2024-04-22T06:26:52.542 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 669 16 3599 3583 0.334687 0 - 0.36018 2024-04-22T06:26:53.542 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 670 16 3599 3583 0.334188 0 - 0.36018 2024-04-22T06:26:54.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 671 16 3599 3583 0.33369 0 - 0.36018 2024-04-22T06:26:55.542 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 672 16 3599 3583 0.333193 0 - 0.36018 2024-04-22T06:26:56.542 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 673 16 3599 3583 0.332698 0 - 0.36018 2024-04-22T06:26:57.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 674 16 3599 3583 0.332205 0 - 0.36018 2024-04-22T06:26:58.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 675 16 3599 3583 0.331712 0 - 0.36018 2024-04-22T06:26:59.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 676 16 3599 3583 0.331222 0 - 0.36018 2024-04-22T06:27:00.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 677 16 3599 3583 0.330732 0 - 0.36018 2024-04-22T06:27:01.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 678 16 3599 3583 0.330245 0 - 0.36018 2024-04-22T06:27:02.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 679 16 3599 3583 0.329758 0 - 0.36018 2024-04-22T06:27:03.543 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:27:03.545715+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:27:03.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:27:03.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 680 16 3599 3583 0.329273 0 - 0.36018 2024-04-22T06:27:04.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 681 16 3599 3583 0.32879 0 - 0.36018 2024-04-22T06:27:05.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 682 16 3599 3583 0.328308 0 - 0.36018 2024-04-22T06:27:06.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 683 16 3599 3583 0.327827 0 - 0.36018 2024-04-22T06:27:07.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 684 16 3599 3583 0.327348 0 - 0.36018 2024-04-22T06:27:08.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 685 16 3599 3583 0.32687 0 - 0.36018 2024-04-22T06:27:09.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 686 16 3599 3583 0.326393 0 - 0.36018 2024-04-22T06:27:10.544 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 687 16 3599 3583 0.325918 0 - 0.36018 2024-04-22T06:27:11.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 688 16 3599 3583 0.325445 0 - 0.36018 2024-04-22T06:27:12.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 689 16 3599 3583 0.324972 0 - 0.36018 2024-04-22T06:27:13.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 690 16 3599 3583 0.324501 0 - 0.36018 2024-04-22T06:27:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:27:13.972 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:27:14.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 691 16 3599 3583 0.324032 0 - 0.36018 2024-04-22T06:27:15.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 692 16 3599 3583 0.323563 0 - 0.36018 2024-04-22T06:27:15.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:27:16.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:27:16.096+0000 7fedc006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:27:16.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 693 16 3599 3583 0.323096 0 - 0.36018 2024-04-22T06:27:17.545 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 694 16 3599 3583 0.322631 0 - 0.36018 2024-04-22T06:27:18.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 695 16 3599 3583 0.322167 0 - 0.36018 2024-04-22T06:27:19.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 696 16 3599 3583 0.321704 0 - 0.36018 2024-04-22T06:27:20.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 697 16 3599 3583 0.321242 0 - 0.36018 2024-04-22T06:27:21.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 698 16 3599 3583 0.320782 0 - 0.36018 2024-04-22T06:27:22.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 699 16 3599 3583 0.320323 0 - 0.36018 2024-04-22T06:27:23.546 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:27:23.548624+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:27:23.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:27:23.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 700 16 3599 3583 0.319866 0 - 0.36018 2024-04-22T06:27:24.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 701 16 3599 3583 0.319409 0 - 0.36018 2024-04-22T06:27:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:27:24.565 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 797, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 766, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2449, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-22T06:27:24.565 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 797, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 766, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2449, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-22T06:27:25.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 702 16 3599 3583 0.318954 0 - 0.36018 2024-04-22T06:27:26.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 703 16 3599 3583 0.3185 0 - 0.36018 2024-04-22T06:27:27.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 704 16 3599 3583 0.318048 0 - 0.36018 2024-04-22T06:27:28.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 705 16 3599 3583 0.317597 0 - 0.36018 2024-04-22T06:27:29.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 706 16 3599 3583 0.317147 0 - 0.36018 2024-04-22T06:27:30.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 707 16 3599 3583 0.316698 0 - 0.36018 2024-04-22T06:27:31.547 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 708 16 3599 3583 0.316251 0 - 0.36018 2024-04-22T06:27:32.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 709 16 3599 3583 0.315805 0 - 0.36018 2024-04-22T06:27:33.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 710 16 3599 3583 0.31536 0 - 0.36018 2024-04-22T06:27:34.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 711 16 3599 3583 0.314917 0 - 0.36018 2024-04-22T06:27:35.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 712 16 3599 3583 0.314474 0 - 0.36018 2024-04-22T06:27:36.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 713 16 3599 3583 0.314033 0 - 0.36018 2024-04-22T06:27:37.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 714 16 3599 3583 0.313594 0 - 0.36018 2024-04-22T06:27:38.548 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 715 16 3599 3583 0.313155 0 - 0.36018 2024-04-22T06:27:39.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 716 16 3599 3583 0.312718 0 - 0.36018 2024-04-22T06:27:40.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 717 16 3599 3583 0.312281 0 - 0.36018 2024-04-22T06:27:41.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 718 16 3599 3583 0.311847 0 - 0.36018 2024-04-22T06:27:42.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 719 16 3599 3583 0.311413 0 - 0.36018 2024-04-22T06:27:43.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:27:43.551541+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:27:43.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:27:43.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 720 16 3599 3583 0.31098 0 - 0.36018 2024-04-22T06:27:44.549 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 721 16 3599 3583 0.310549 0 - 0.36018 2024-04-22T06:27:45.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 722 16 3599 3583 0.310119 0 - 0.36018 2024-04-22T06:27:46.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 723 16 3599 3583 0.30969 0 - 0.36018 2024-04-22T06:27:47.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 724 16 3599 3583 0.309262 0 - 0.36018 2024-04-22T06:27:48.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 725 16 3599 3583 0.308836 0 - 0.36018 2024-04-22T06:27:49.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 726 16 3599 3583 0.30841 0 - 0.36018 2024-04-22T06:27:50.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 727 16 3599 3583 0.307986 0 - 0.36018 2024-04-22T06:27:51.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 728 16 3599 3583 0.307563 0 - 0.36018 2024-04-22T06:27:52.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 729 16 3599 3583 0.307141 0 - 0.36018 2024-04-22T06:27:53.550 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 730 16 3599 3583 0.30672 0 - 0.36018 2024-04-22T06:27:54.551 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 731 16 3599 3583 0.306301 0 - 0.36018 2024-04-22T06:27:55.551 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 732 16 3599 3583 0.305882 0 - 0.36018 2024-04-22T06:27:56.551 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 733 16 3599 3583 0.305465 0 - 0.36018 2024-04-22T06:27:57.551 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 734 16 3599 3583 0.305049 0 - 0.36018 2024-04-22T06:27:58.551 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 735 16 3599 3583 0.304634 0 - 0.36018 2024-04-22T06:27:59.551 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 736 16 3599 3583 0.30422 0 - 0.36018 2024-04-22T06:28:00.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 737 16 3599 3583 0.303807 0 - 0.36018 2024-04-22T06:28:01.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 738 16 3599 3583 0.303395 0 - 0.36018 2024-04-22T06:28:02.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 739 16 3599 3583 0.302985 0 - 0.36018 2024-04-22T06:28:03.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:28:03.554251+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:28:03.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:28:03.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 740 16 3599 3583 0.302575 0 - 0.36018 2024-04-22T06:28:04.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 741 16 3599 3583 0.302167 0 - 0.36018 2024-04-22T06:28:05.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 742 16 3599 3583 0.30176 0 - 0.36018 2024-04-22T06:28:06.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 743 16 3599 3583 0.301354 0 - 0.36018 2024-04-22T06:28:07.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 744 16 3599 3583 0.300949 0 - 0.36018 2024-04-22T06:28:08.552 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 745 16 3599 3583 0.300545 0 - 0.36018 2024-04-22T06:28:09.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 746 16 3599 3583 0.300142 0 - 0.36018 2024-04-22T06:28:10.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 747 16 3599 3583 0.29974 0 - 0.36018 2024-04-22T06:28:11.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 748 16 3599 3583 0.299339 0 - 0.36018 2024-04-22T06:28:12.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 749 16 3599 3583 0.29894 0 - 0.36018 2024-04-22T06:28:13.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 750 16 3599 3583 0.298541 0 - 0.36018 2024-04-22T06:28:14.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 751 16 3599 3583 0.298144 0 - 0.36018 2024-04-22T06:28:15.553 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 752 16 3599 3583 0.297747 0 - 0.36018 2024-04-22T06:28:16.554 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 753 16 3599 3583 0.297352 0 - 0.36018 2024-04-22T06:28:17.554 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 754 16 3599 3583 0.296957 0 - 0.36018 2024-04-22T06:28:18.554 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 755 16 3599 3583 0.296564 0 - 0.36018 2024-04-22T06:28:19.554 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 756 16 3599 3583 0.296172 0 - 0.36018 2024-04-22T06:28:20.554 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 757 16 3599 3583 0.295781 0 - 0.36018 2024-04-22T06:28:21.554 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 758 16 3599 3583 0.29539 0 - 0.36018 2024-04-22T06:28:22.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 759 16 3599 3583 0.295001 0 - 0.36018 2024-04-22T06:28:23.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:28:23.557105+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:28:23.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:28:23.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 760 16 3599 3583 0.294613 0 - 0.36018 2024-04-22T06:28:24.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 761 16 3599 3583 0.294226 0 - 0.36018 2024-04-22T06:28:25.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 762 16 3599 3583 0.29384 0 - 0.36018 2024-04-22T06:28:26.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 763 16 3599 3583 0.293455 0 - 0.36018 2024-04-22T06:28:27.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 764 16 3599 3583 0.29307 0 - 0.36018 2024-04-22T06:28:28.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 765 16 3599 3583 0.292687 0 - 0.36018 2024-04-22T06:28:29.555 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 766 16 3599 3583 0.292305 0 - 0.36018 2024-04-22T06:28:30.556 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 767 16 3599 3583 0.291924 0 - 0.36018 2024-04-22T06:28:31.556 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 768 16 3599 3583 0.291544 0 - 0.36018 2024-04-22T06:28:32.556 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 769 16 3599 3583 0.291165 0 - 0.36018 2024-04-22T06:28:33.556 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 770 16 3599 3583 0.290787 0 - 0.36018 2024-04-22T06:28:34.556 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 771 16 3599 3583 0.29041 0 - 0.36018 2024-04-22T06:28:35.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 772 16 3599 3583 0.290033 0 - 0.36018 2024-04-22T06:28:36.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 773 16 3599 3583 0.289658 0 - 0.36018 2024-04-22T06:28:37.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 774 16 3599 3583 0.289284 0 - 0.36018 2024-04-22T06:28:38.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 775 16 3599 3583 0.288911 0 - 0.36018 2024-04-22T06:28:39.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 776 16 3599 3583 0.288538 0 - 0.36018 2024-04-22T06:28:40.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 777 16 3599 3583 0.288167 0 - 0.36018 2024-04-22T06:28:41.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 778 16 3599 3583 0.287797 0 - 0.36018 2024-04-22T06:28:42.557 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 779 16 3599 3583 0.287427 0 - 0.36018 2024-04-22T06:28:43.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:28:43.560072+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:28:43.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:28:43.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 780 16 3599 3583 0.287059 0 - 0.36018 2024-04-22T06:28:44.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 781 16 3599 3583 0.286691 0 - 0.36018 2024-04-22T06:28:45.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 782 16 3599 3583 0.286325 0 - 0.36018 2024-04-22T06:28:46.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 783 16 3599 3583 0.285959 0 - 0.36018 2024-04-22T06:28:47.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 784 16 3599 3583 0.285594 0 - 0.36018 2024-04-22T06:28:48.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 785 16 3599 3583 0.28523 0 - 0.36018 2024-04-22T06:28:49.558 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 786 16 3599 3583 0.284867 0 - 0.36018 2024-04-22T06:28:50.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 787 16 3599 3583 0.284505 0 - 0.36018 2024-04-22T06:28:51.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 788 16 3599 3583 0.284144 0 - 0.36018 2024-04-22T06:28:52.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 789 16 3599 3583 0.283784 0 - 0.36018 2024-04-22T06:28:53.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 790 16 3599 3583 0.283425 0 - 0.36018 2024-04-22T06:28:54.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 791 16 3599 3583 0.283067 0 - 0.36018 2024-04-22T06:28:55.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 792 16 3599 3583 0.282709 0 - 0.36018 2024-04-22T06:28:56.559 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 793 16 3599 3583 0.282353 0 - 0.36018 2024-04-22T06:28:57.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 794 16 3599 3583 0.281997 0 - 0.36018 2024-04-22T06:28:58.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 795 16 3599 3583 0.281643 0 - 0.36018 2024-04-22T06:28:59.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 796 16 3599 3583 0.281289 0 - 0.36018 2024-04-22T06:29:00.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 797 16 3599 3583 0.280936 0 - 0.36018 2024-04-22T06:29:01.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 798 16 3599 3583 0.280584 0 - 0.36018 2024-04-22T06:29:02.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 799 16 3599 3583 0.280233 0 - 0.36018 2024-04-22T06:29:03.560 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:29:03.562956+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:29:03.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:29:03.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 800 16 3599 3583 0.279882 0 - 0.36018 2024-04-22T06:29:04.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 801 16 3599 3583 0.279533 0 - 0.36018 2024-04-22T06:29:05.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 802 16 3599 3583 0.279184 0 - 0.36018 2024-04-22T06:29:06.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 803 16 3599 3583 0.278837 0 - 0.36018 2024-04-22T06:29:07.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 804 16 3599 3583 0.27849 0 - 0.36018 2024-04-22T06:29:08.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 805 16 3599 3583 0.278144 0 - 0.36018 2024-04-22T06:29:09.561 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 806 16 3599 3583 0.277799 0 - 0.36018 2024-04-22T06:29:10.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 807 16 3599 3583 0.277455 0 - 0.36018 2024-04-22T06:29:11.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 808 16 3599 3583 0.277111 0 - 0.36018 2024-04-22T06:29:12.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 809 16 3599 3583 0.276769 0 - 0.36018 2024-04-22T06:29:13.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 810 16 3599 3583 0.276427 0 - 0.36018 2024-04-22T06:29:14.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 811 16 3599 3583 0.276086 0 - 0.36018 2024-04-22T06:29:15.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 812 16 3599 3583 0.275746 0 - 0.36018 2024-04-22T06:29:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:29:16.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:29:16.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 813 16 3599 3583 0.275407 0 - 0.36018 2024-04-22T06:29:17.562 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 814 16 3599 3583 0.275069 0 - 0.36018 2024-04-22T06:29:18.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:29:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:29:18.133+0000 7f970dc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:29:18.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 815 16 3599 3583 0.274731 0 - 0.36018 2024-04-22T06:29:19.563 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 816 16 3599 3583 0.274394 0 - 0.36018 2024-04-22T06:29:20.563 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 817 16 3599 3583 0.274058 0 - 0.36018 2024-04-22T06:29:21.563 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 818 16 3599 3583 0.273723 0 - 0.36018 2024-04-22T06:29:22.563 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 819 16 3599 3583 0.273389 0 - 0.36018 2024-04-22T06:29:23.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:29:23.565836+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:29:23.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:29:23.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 820 16 3599 3583 0.273056 0 - 0.36018 2024-04-22T06:29:24.563 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 821 16 3599 3583 0.272723 0 - 0.36018 2024-04-22T06:29:25.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 822 16 3599 3583 0.272391 0 - 0.36018 2024-04-22T06:29:26.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 823 16 3599 3583 0.27206 0 - 0.36018 2024-04-22T06:29:27.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 824 16 3599 3583 0.27173 0 - 0.36018 2024-04-22T06:29:28.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 825 16 3599 3583 0.271401 0 - 0.36018 2024-04-22T06:29:29.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 826 16 3599 3583 0.271072 0 - 0.36018 2024-04-22T06:29:30.564 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 827 16 3599 3583 0.270745 0 - 0.36018 2024-04-22T06:29:31.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 828 16 3599 3583 0.270418 0 - 0.36018 2024-04-22T06:29:32.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 829 16 3599 3583 0.270091 0 - 0.36018 2024-04-22T06:29:33.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 830 16 3599 3583 0.269766 0 - 0.36018 2024-04-22T06:29:34.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 831 16 3599 3583 0.269441 0 - 0.36018 2024-04-22T06:29:35.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 832 16 3599 3583 0.269117 0 - 0.36018 2024-04-22T06:29:36.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 833 16 3599 3583 0.268794 0 - 0.36018 2024-04-22T06:29:37.565 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 834 16 3599 3583 0.268472 0 - 0.36018 2024-04-22T06:29:38.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 835 16 3599 3583 0.268151 0 - 0.36018 2024-04-22T06:29:39.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 836 16 3599 3583 0.26783 0 - 0.36018 2024-04-22T06:29:40.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 837 16 3599 3583 0.26751 0 - 0.36018 2024-04-22T06:29:41.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 838 16 3599 3583 0.267191 0 - 0.36018 2024-04-22T06:29:42.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 839 16 3599 3583 0.266872 0 - 0.36018 2024-04-22T06:29:43.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:29:43.568914+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:29:43.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:29:43.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 840 16 3599 3583 0.266554 0 - 0.36018 2024-04-22T06:29:44.566 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 841 16 3599 3583 0.266237 0 - 0.36018 2024-04-22T06:29:45.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 842 16 3599 3583 0.265921 0 - 0.36018 2024-04-22T06:29:46.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 843 16 3599 3583 0.265606 0 - 0.36018 2024-04-22T06:29:47.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 844 16 3599 3583 0.265291 0 - 0.36018 2024-04-22T06:29:48.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 845 16 3599 3583 0.264977 0 - 0.36018 2024-04-22T06:29:49.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 846 16 3599 3583 0.264664 0 - 0.36018 2024-04-22T06:29:50.567 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 847 16 3599 3583 0.264351 0 - 0.36018 2024-04-22T06:29:51.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 848 16 3599 3583 0.26404 0 - 0.36018 2024-04-22T06:29:52.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 849 16 3599 3583 0.263729 0 - 0.36018 2024-04-22T06:29:53.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 850 16 3599 3583 0.263418 0 - 0.36018 2024-04-22T06:29:54.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 851 16 3599 3583 0.263109 0 - 0.36018 2024-04-22T06:29:55.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 852 16 3599 3583 0.2628 0 - 0.36018 2024-04-22T06:29:56.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 853 16 3599 3583 0.262492 0 - 0.36018 2024-04-22T06:29:57.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 854 16 3599 3583 0.262185 0 - 0.36018 2024-04-22T06:29:58.568 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 855 16 3599 3583 0.261878 0 - 0.36018 2024-04-22T06:29:59.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 856 16 3599 3583 0.261572 0 - 0.36018 2024-04-22T06:30:00.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 857 16 3599 3583 0.261267 0 - 0.36018 2024-04-22T06:30:01.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 858 16 3599 3583 0.260962 0 - 0.36018 2024-04-22T06:30:02.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 859 16 3599 3583 0.260659 0 - 0.36018 2024-04-22T06:30:03.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:30:03.571849+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:30:03.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:30:03.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 860 16 3599 3583 0.260355 0 - 0.36018 2024-04-22T06:30:04.569 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 861 16 3599 3583 0.260053 0 - 0.36018 2024-04-22T06:30:05.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 862 16 3599 3583 0.259751 0 - 0.36018 2024-04-22T06:30:06.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 863 16 3599 3583 0.25945 0 - 0.36018 2024-04-22T06:30:07.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 864 16 3599 3583 0.25915 0 - 0.36018 2024-04-22T06:30:08.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 865 16 3599 3583 0.258851 0 - 0.36018 2024-04-22T06:30:09.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 866 16 3599 3583 0.258552 0 - 0.36018 2024-04-22T06:30:10.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 867 16 3599 3583 0.258253 0 - 0.36018 2024-04-22T06:30:11.570 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 868 16 3599 3583 0.257956 0 - 0.36018 2024-04-22T06:30:12.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 869 16 3599 3583 0.257659 0 - 0.36018 2024-04-22T06:30:13.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 870 16 3599 3583 0.257363 0 - 0.36018 2024-04-22T06:30:14.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 871 16 3599 3583 0.257067 0 - 0.36018 2024-04-22T06:30:15.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 872 16 3599 3583 0.256773 0 - 0.36018 2024-04-22T06:30:16.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 873 16 3599 3583 0.256478 0 - 0.36018 2024-04-22T06:30:17.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 874 16 3599 3583 0.256185 0 - 0.36018 2024-04-22T06:30:18.571 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 875 16 3599 3583 0.255892 0 - 0.36018 2024-04-22T06:30:19.572 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 876 16 3599 3583 0.2556 0 - 0.36018 2024-04-22T06:30:20.572 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 877 16 3599 3583 0.255309 0 - 0.36018 2024-04-22T06:30:21.572 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 878 16 3599 3583 0.255018 0 - 0.36018 2024-04-22T06:30:22.572 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 879 16 3599 3583 0.254728 0 - 0.36018 2024-04-22T06:30:23.572 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:30:23.574824+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:30:23.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:30:23.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 880 16 3599 3583 0.254438 0 - 0.36018 2024-04-22T06:30:24.572 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 881 16 3599 3583 0.254149 0 - 0.36018 2024-04-22T06:30:25.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 882 16 3599 3583 0.253861 0 - 0.36018 2024-04-22T06:30:26.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 883 16 3599 3583 0.253574 0 - 0.36018 2024-04-22T06:30:27.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 884 16 3599 3583 0.253287 0 - 0.36018 2024-04-22T06:30:28.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 885 16 3599 3583 0.253001 0 - 0.36018 2024-04-22T06:30:29.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 886 16 3599 3583 0.252715 0 - 0.36018 2024-04-22T06:30:30.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 887 16 3599 3583 0.25243 0 - 0.36018 2024-04-22T06:30:31.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 888 16 3599 3583 0.252146 0 - 0.36018 2024-04-22T06:30:32.573 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 889 16 3599 3583 0.251862 0 - 0.36018 2024-04-22T06:30:33.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 890 16 3599 3583 0.251579 0 - 0.36018 2024-04-22T06:30:34.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 891 16 3599 3583 0.251297 0 - 0.36018 2024-04-22T06:30:35.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 892 16 3599 3583 0.251015 0 - 0.36018 2024-04-22T06:30:36.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 893 16 3599 3583 0.250734 0 - 0.36018 2024-04-22T06:30:37.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 894 16 3599 3583 0.250454 0 - 0.36018 2024-04-22T06:30:38.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 895 16 3599 3583 0.250174 0 - 0.36018 2024-04-22T06:30:39.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 896 16 3599 3583 0.249895 0 - 0.36018 2024-04-22T06:30:40.574 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 897 16 3599 3583 0.249616 0 - 0.36018 2024-04-22T06:30:41.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 898 16 3599 3583 0.249338 0 - 0.36018 2024-04-22T06:30:42.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 899 16 3599 3583 0.249061 0 - 0.36018 2024-04-22T06:30:43.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:30:43.577762+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:30:43.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:30:43.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 900 16 3599 3583 0.248784 0 - 0.36018 2024-04-22T06:30:44.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 901 16 3599 3583 0.248508 0 - 0.36018 2024-04-22T06:30:45.575 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 902 16 3599 3583 0.248232 0 - 0.36018 2024-04-22T06:30:46.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 903 16 3599 3583 0.247958 0 - 0.36018 2024-04-22T06:30:47.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 904 16 3599 3583 0.247683 0 - 0.36018 2024-04-22T06:30:48.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 905 16 3599 3583 0.24741 0 - 0.36018 2024-04-22T06:30:49.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 906 16 3599 3583 0.247136 0 - 0.36018 2024-04-22T06:30:50.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 907 16 3599 3583 0.246864 0 - 0.36018 2024-04-22T06:30:51.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 908 16 3599 3583 0.246592 0 - 0.36018 2024-04-22T06:30:52.576 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 909 16 3599 3583 0.246321 0 - 0.36018 2024-04-22T06:30:53.577 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 910 16 3599 3583 0.24605 0 - 0.36018 2024-04-22T06:30:54.577 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 911 16 3599 3583 0.24578 0 - 0.36018 2024-04-22T06:30:55.577 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 912 16 3599 3583 0.245511 0 - 0.36018 2024-04-22T06:30:56.577 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 913 16 3599 3583 0.245242 0 - 0.36018 2024-04-22T06:30:57.577 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 914 16 3599 3583 0.244973 0 - 0.36018 2024-04-22T06:30:58.577 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 915 16 3599 3583 0.244706 0 - 0.36018 2024-04-22T06:30:59.578 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 916 16 3599 3583 0.244438 0 - 0.36018 2024-04-22T06:31:00.578 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 917 16 3599 3583 0.244172 0 - 0.36018 2024-04-22T06:31:01.578 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 918 16 3599 3583 0.243906 0 - 0.36018 2024-04-22T06:31:02.578 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 919 16 3599 3583 0.24364 0 - 0.36018 2024-04-22T06:31:03.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:31:03.580845+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:31:03.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:31:03.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 920 16 3599 3583 0.243376 0 - 0.36018 2024-04-22T06:31:04.578 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 921 16 3599 3583 0.243111 0 - 0.36018 2024-04-22T06:31:05.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 922 16 3599 3583 0.242848 0 - 0.36018 2024-04-22T06:31:06.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 923 16 3599 3583 0.242585 0 - 0.36018 2024-04-22T06:31:07.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 924 16 3599 3583 0.242322 0 - 0.36018 2024-04-22T06:31:08.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 925 16 3599 3583 0.24206 0 - 0.36018 2024-04-22T06:31:09.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 926 16 3599 3583 0.241799 0 - 0.36018 2024-04-22T06:31:10.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 927 16 3599 3583 0.241538 0 - 0.36018 2024-04-22T06:31:11.579 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 928 16 3599 3583 0.241278 0 - 0.36018 2024-04-22T06:31:12.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 929 16 3599 3583 0.241018 0 - 0.36018 2024-04-22T06:31:13.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 930 16 3599 3583 0.240759 0 - 0.36018 2024-04-22T06:31:14.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 931 16 3599 3583 0.2405 0 - 0.36018 2024-04-22T06:31:15.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 932 16 3599 3583 0.240242 0 - 0.36018 2024-04-22T06:31:16.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 933 16 3599 3583 0.239985 0 - 0.36018 2024-04-22T06:31:17.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 934 16 3599 3583 0.239728 0 - 0.36018 2024-04-22T06:31:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:31:18.047 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:31:18.580 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 935 16 3599 3583 0.239471 0 - 0.36018 2024-04-22T06:31:19.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 936 16 3599 3583 0.239215 0 - 0.36018 2024-04-22T06:31:20.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:31:20.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:31:20.171+0000 7ff178aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:31:20.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 937 16 3599 3583 0.23896 0 - 0.36018 2024-04-22T06:31:21.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 938 16 3599 3583 0.238705 0 - 0.36018 2024-04-22T06:31:22.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 939 16 3599 3583 0.238451 0 - 0.36018 2024-04-22T06:31:23.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:31:23.583823+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:31:23.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:31:23.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 940 16 3599 3583 0.238197 0 - 0.36018 2024-04-22T06:31:24.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 941 16 3599 3583 0.237944 0 - 0.36018 2024-04-22T06:31:25.581 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 942 16 3599 3583 0.237692 0 - 0.36018 2024-04-22T06:31:26.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 943 16 3599 3583 0.23744 0 - 0.36018 2024-04-22T06:31:27.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 944 16 3599 3583 0.237188 0 - 0.36018 2024-04-22T06:31:28.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 945 16 3599 3583 0.236937 0 - 0.36018 2024-04-22T06:31:29.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 946 16 3599 3583 0.236687 0 - 0.36018 2024-04-22T06:31:30.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 947 16 3599 3583 0.236437 0 - 0.36018 2024-04-22T06:31:31.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 948 16 3599 3583 0.236187 0 - 0.36018 2024-04-22T06:31:32.582 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 949 16 3599 3583 0.235938 0 - 0.36018 2024-04-22T06:31:33.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 950 16 3599 3583 0.23569 0 - 0.36018 2024-04-22T06:31:34.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 951 16 3599 3583 0.235442 0 - 0.36018 2024-04-22T06:31:35.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 952 16 3599 3583 0.235195 0 - 0.36018 2024-04-22T06:31:36.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 953 16 3599 3583 0.234948 0 - 0.36018 2024-04-22T06:31:37.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 954 16 3599 3583 0.234702 0 - 0.36018 2024-04-22T06:31:38.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 955 16 3599 3583 0.234456 0 - 0.36018 2024-04-22T06:31:39.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 956 16 3599 3583 0.234211 0 - 0.36018 2024-04-22T06:31:40.583 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 957 16 3599 3583 0.233966 0 - 0.36018 2024-04-22T06:31:41.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 958 16 3599 3583 0.233722 0 - 0.36018 2024-04-22T06:31:42.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 959 16 3599 3583 0.233478 0 - 0.36018 2024-04-22T06:31:43.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:31:43.586669+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:31:43.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:31:43.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 960 16 3599 3583 0.233235 0 - 0.36018 2024-04-22T06:31:44.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 961 16 3599 3583 0.232992 0 - 0.36018 2024-04-22T06:31:45.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 962 16 3599 3583 0.23275 0 - 0.36018 2024-04-22T06:31:46.584 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 963 16 3599 3583 0.232508 0 - 0.36018 2024-04-22T06:31:47.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 964 16 3599 3583 0.232267 0 - 0.36018 2024-04-22T06:31:48.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 965 16 3599 3583 0.232026 0 - 0.36018 2024-04-22T06:31:49.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 966 16 3599 3583 0.231786 0 - 0.36018 2024-04-22T06:31:50.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 967 16 3599 3583 0.231547 0 - 0.36018 2024-04-22T06:31:51.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 968 16 3599 3583 0.231307 0 - 0.36018 2024-04-22T06:31:52.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 969 16 3599 3583 0.231069 0 - 0.36018 2024-04-22T06:31:53.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 970 16 3599 3583 0.23083 0 - 0.36018 2024-04-22T06:31:54.585 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 971 16 3599 3583 0.230593 0 - 0.36018 2024-04-22T06:31:55.586 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 972 16 3599 3583 0.230356 0 - 0.36018 2024-04-22T06:31:56.586 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 973 16 3599 3583 0.230119 0 - 0.36018 2024-04-22T06:31:57.586 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 974 16 3599 3583 0.229883 0 - 0.36018 2024-04-22T06:31:58.586 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 975 16 3599 3583 0.229647 0 - 0.36018 2024-04-22T06:31:59.586 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 976 16 3599 3583 0.229411 0 - 0.36018 2024-04-22T06:32:00.586 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 977 16 3599 3583 0.229177 0 - 0.36018 2024-04-22T06:32:01.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 978 16 3599 3583 0.228942 0 - 0.36018 2024-04-22T06:32:02.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 979 16 3599 3583 0.228708 0 - 0.36018 2024-04-22T06:32:03.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:32:03.589500+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:32:03.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:32:03.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 980 16 3599 3583 0.228475 0 - 0.36018 2024-04-22T06:32:04.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 981 16 3599 3583 0.228242 0 - 0.36018 2024-04-22T06:32:05.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 982 16 3599 3583 0.22801 0 - 0.36018 2024-04-22T06:32:06.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 983 16 3599 3583 0.227778 0 - 0.36018 2024-04-22T06:32:07.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 984 16 3599 3583 0.227546 0 - 0.36018 2024-04-22T06:32:08.587 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 985 16 3599 3583 0.227315 0 - 0.36018 2024-04-22T06:32:09.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 986 16 3599 3583 0.227085 0 - 0.36018 2024-04-22T06:32:10.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 987 16 3599 3583 0.226855 0 - 0.36018 2024-04-22T06:32:11.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 988 16 3599 3583 0.226625 0 - 0.36018 2024-04-22T06:32:12.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 989 16 3599 3583 0.226396 0 - 0.36018 2024-04-22T06:32:13.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 990 16 3599 3583 0.226167 0 - 0.36018 2024-04-22T06:32:14.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 991 16 3599 3583 0.225939 0 - 0.36018 2024-04-22T06:32:15.588 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 992 16 3599 3583 0.225711 0 - 0.36018 2024-04-22T06:32:16.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 993 16 3599 3583 0.225484 0 - 0.36018 2024-04-22T06:32:17.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 994 16 3599 3583 0.225257 0 - 0.36018 2024-04-22T06:32:18.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 995 16 3599 3583 0.225031 0 - 0.36018 2024-04-22T06:32:19.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 996 16 3599 3583 0.224805 0 - 0.36018 2024-04-22T06:32:20.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 997 16 3599 3583 0.224579 0 - 0.36018 2024-04-22T06:32:21.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 998 16 3599 3583 0.224354 0 - 0.36018 2024-04-22T06:32:22.589 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 999 16 3599 3583 0.22413 0 - 0.36018 2024-04-22T06:32:23.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:32:23.592398+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:32:23.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:32:23.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1000 16 3599 3583 0.223906 0 - 0.36018 2024-04-22T06:32:24.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1001 16 3599 3583 0.223682 0 - 0.36018 2024-04-22T06:32:25.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1002 16 3599 3583 0.223459 0 - 0.36018 2024-04-22T06:32:26.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1003 16 3599 3583 0.223236 0 - 0.36018 2024-04-22T06:32:27.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1004 16 3599 3583 0.223013 0 - 0.36018 2024-04-22T06:32:28.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1005 16 3599 3583 0.222792 0 - 0.36018 2024-04-22T06:32:29.590 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1006 16 3599 3583 0.22257 0 - 0.36018 2024-04-22T06:32:30.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1007 16 3599 3583 0.222349 0 - 0.36018 2024-04-22T06:32:31.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1008 16 3599 3583 0.222129 0 - 0.36018 2024-04-22T06:32:32.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1009 16 3599 3583 0.221908 0 - 0.36018 2024-04-22T06:32:33.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1010 16 3599 3583 0.221689 0 - 0.36018 2024-04-22T06:32:34.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1011 16 3599 3583 0.221469 0 - 0.36018 2024-04-22T06:32:35.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1012 16 3599 3583 0.221251 0 - 0.36018 2024-04-22T06:32:36.591 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1013 16 3599 3583 0.221032 0 - 0.36018 2024-04-22T06:32:37.592 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1014 16 3599 3583 0.220814 0 - 0.36018 2024-04-22T06:32:38.592 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1015 16 3599 3583 0.220597 0 - 0.36018 2024-04-22T06:32:39.592 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1016 16 3599 3583 0.220379 0 - 0.36018 2024-04-22T06:32:40.592 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1017 16 3599 3583 0.220163 0 - 0.36018 2024-04-22T06:32:41.592 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1018 16 3599 3583 0.219946 0 - 0.36018 2024-04-22T06:32:42.592 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1019 16 3599 3583 0.219731 0 - 0.36018 2024-04-22T06:32:43.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:32:43.595329+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:32:43.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:32:43.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1020 16 3599 3583 0.219515 0 - 0.36018 2024-04-22T06:32:44.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1021 16 3599 3583 0.2193 0 - 0.36018 2024-04-22T06:32:45.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1022 16 3599 3583 0.219086 0 - 0.36018 2024-04-22T06:32:46.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1023 16 3599 3583 0.218871 0 - 0.36018 2024-04-22T06:32:47.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1024 16 3599 3583 0.218658 0 - 0.36018 2024-04-22T06:32:48.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1025 16 3599 3583 0.218444 0 - 0.36018 2024-04-22T06:32:49.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1026 16 3599 3583 0.218232 0 - 0.36018 2024-04-22T06:32:50.593 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1027 16 3599 3583 0.218019 0 - 0.36018 2024-04-22T06:32:51.594 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1028 16 3599 3583 0.217807 0 - 0.36018 2024-04-22T06:32:52.594 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1029 16 3599 3583 0.217595 0 - 0.36018 2024-04-22T06:32:53.594 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1030 16 3599 3583 0.217384 0 - 0.36018 2024-04-22T06:32:54.594 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1031 16 3599 3583 0.217173 0 - 0.36018 2024-04-22T06:32:55.594 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1032 16 3599 3583 0.216963 0 - 0.36018 2024-04-22T06:32:56.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1033 16 3599 3583 0.216753 0 - 0.36018 2024-04-22T06:32:57.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1034 16 3599 3583 0.216543 0 - 0.36018 2024-04-22T06:32:58.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1035 16 3599 3583 0.216334 0 - 0.36018 2024-04-22T06:32:59.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1036 16 3599 3583 0.216125 0 - 0.36018 2024-04-22T06:33:00.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1037 16 3599 3583 0.215917 0 - 0.36018 2024-04-22T06:33:01.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1038 16 3599 3583 0.215709 0 - 0.36018 2024-04-22T06:33:02.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1039 16 3599 3583 0.215501 0 - 0.36018 2024-04-22T06:33:03.595 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:33:03.598326+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:33:03.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:33:03.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1040 16 3599 3583 0.215294 0 - 0.36018 2024-04-22T06:33:04.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1041 16 3599 3583 0.215087 0 - 0.36018 2024-04-22T06:33:05.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1042 16 3599 3583 0.214881 0 - 0.36018 2024-04-22T06:33:06.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1043 16 3599 3583 0.214674 0 - 0.36018 2024-04-22T06:33:07.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1044 16 3599 3583 0.214469 0 - 0.36018 2024-04-22T06:33:08.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1045 16 3599 3583 0.214264 0 - 0.36018 2024-04-22T06:33:09.596 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1046 16 3599 3583 0.214059 0 - 0.36018 2024-04-22T06:33:10.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1047 16 3599 3583 0.213854 0 - 0.36018 2024-04-22T06:33:11.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1048 16 3599 3583 0.21365 0 - 0.36018 2024-04-22T06:33:12.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1049 16 3599 3583 0.213447 0 - 0.36018 2024-04-22T06:33:13.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1050 16 3599 3583 0.213243 0 - 0.36018 2024-04-22T06:33:14.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1051 16 3599 3583 0.21304 0 - 0.36018 2024-04-22T06:33:15.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1052 16 3599 3583 0.212838 0 - 0.36018 2024-04-22T06:33:16.597 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1053 16 3599 3583 0.212636 0 - 0.36018 2024-04-22T06:33:17.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1054 16 3599 3583 0.212434 0 - 0.36018 2024-04-22T06:33:18.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1055 16 3599 3583 0.212233 0 - 0.36018 2024-04-22T06:33:19.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1056 16 3599 3583 0.212032 0 - 0.36018 2024-04-22T06:33:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:33:20.085 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:33:20.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1057 16 3599 3583 0.211831 0 - 0.36018 2024-04-22T06:33:21.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1058 16 3599 3583 0.211631 0 - 0.36018 2024-04-22T06:33:22.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:33:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:33:22.209+0000 7fb913d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:33:22.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1059 16 3599 3583 0.211431 0 - 0.36018 2024-04-22T06:33:23.598 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:33:23.601235+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:33:23.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:33:23.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1060 16 3599 3583 0.211232 0 - 0.36018 2024-04-22T06:33:24.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1061 16 3599 3583 0.211033 0 - 0.36018 2024-04-22T06:33:25.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1062 16 3599 3583 0.210834 0 - 0.36018 2024-04-22T06:33:26.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1063 16 3599 3583 0.210635 0 - 0.36018 2024-04-22T06:33:27.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1064 16 3599 3583 0.210437 0 - 0.36018 2024-04-22T06:33:28.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1065 16 3599 3583 0.21024 0 - 0.36018 2024-04-22T06:33:29.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1066 16 3599 3583 0.210043 0 - 0.36018 2024-04-22T06:33:30.599 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1067 16 3599 3583 0.209846 0 - 0.36018 2024-04-22T06:33:31.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1068 16 3599 3583 0.209649 0 - 0.36018 2024-04-22T06:33:32.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1069 16 3599 3583 0.209453 0 - 0.36018 2024-04-22T06:33:33.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1070 16 3599 3583 0.209257 0 - 0.36018 2024-04-22T06:33:34.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1071 16 3599 3583 0.209062 0 - 0.36018 2024-04-22T06:33:35.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1072 16 3599 3583 0.208867 0 - 0.36018 2024-04-22T06:33:36.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1073 16 3599 3583 0.208672 0 - 0.36018 2024-04-22T06:33:37.600 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1074 16 3599 3583 0.208478 0 - 0.36018 2024-04-22T06:33:38.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1075 16 3599 3583 0.208284 0 - 0.36018 2024-04-22T06:33:39.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1076 16 3599 3583 0.208091 0 - 0.36018 2024-04-22T06:33:40.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1077 16 3599 3583 0.207897 0 - 0.36018 2024-04-22T06:33:41.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1078 16 3599 3583 0.207705 0 - 0.36018 2024-04-22T06:33:42.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1079 16 3599 3583 0.207512 0 - 0.36018 2024-04-22T06:33:43.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:33:43.604019+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:33:43.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:33:43.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1080 16 3599 3583 0.20732 0 - 0.36018 2024-04-22T06:33:44.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1081 16 3599 3583 0.207128 0 - 0.36018 2024-04-22T06:33:45.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1082 16 3599 3583 0.206937 0 - 0.36018 2024-04-22T06:33:46.601 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1083 16 3599 3583 0.206746 0 - 0.36018 2024-04-22T06:33:47.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1084 16 3599 3583 0.206555 0 - 0.36018 2024-04-22T06:33:48.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1085 16 3599 3583 0.206365 0 - 0.36018 2024-04-22T06:33:49.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1086 16 3599 3583 0.206175 0 - 0.36018 2024-04-22T06:33:50.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1087 16 3599 3583 0.205985 0 - 0.36018 2024-04-22T06:33:51.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1088 16 3599 3583 0.205796 0 - 0.36018 2024-04-22T06:33:52.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1089 16 3599 3583 0.205607 0 - 0.36018 2024-04-22T06:33:53.602 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1090 16 3599 3583 0.205418 0 - 0.36018 2024-04-22T06:33:54.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1091 16 3599 3583 0.20523 0 - 0.36018 2024-04-22T06:33:55.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1092 16 3599 3583 0.205042 0 - 0.36018 2024-04-22T06:33:56.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1093 16 3599 3583 0.204854 0 - 0.36018 2024-04-22T06:33:57.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1094 16 3599 3583 0.204667 0 - 0.36018 2024-04-22T06:33:58.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1095 16 3599 3583 0.20448 0 - 0.36018 2024-04-22T06:33:59.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1096 16 3599 3583 0.204293 0 - 0.36018 2024-04-22T06:34:00.603 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1097 16 3599 3583 0.204107 0 - 0.36018 2024-04-22T06:34:01.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1098 16 3599 3583 0.203921 0 - 0.36018 2024-04-22T06:34:02.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1099 16 3599 3583 0.203736 0 - 0.36018 2024-04-22T06:34:03.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:34:03.606794+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:34:03.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:34:03.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1100 16 3599 3583 0.20355 0 - 0.36018 2024-04-22T06:34:04.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1101 16 3599 3583 0.203366 0 - 0.36018 2024-04-22T06:34:05.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1102 16 3599 3583 0.203181 0 - 0.36018 2024-04-22T06:34:06.604 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1103 16 3599 3583 0.202997 0 - 0.36018 2024-04-22T06:34:07.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1104 16 3599 3583 0.202813 0 - 0.36018 2024-04-22T06:34:08.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1105 16 3599 3583 0.202629 0 - 0.36018 2024-04-22T06:34:09.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1106 16 3599 3583 0.202446 0 - 0.36018 2024-04-22T06:34:10.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1107 16 3599 3583 0.202263 0 - 0.36018 2024-04-22T06:34:11.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1108 16 3599 3583 0.202081 0 - 0.36018 2024-04-22T06:34:12.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1109 16 3599 3583 0.201899 0 - 0.36018 2024-04-22T06:34:13.605 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1110 16 3599 3583 0.201717 0 - 0.36018 2024-04-22T06:34:14.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1111 16 3599 3583 0.201535 0 - 0.36018 2024-04-22T06:34:15.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1112 16 3599 3583 0.201354 0 - 0.36018 2024-04-22T06:34:16.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1113 16 3599 3583 0.201173 0 - 0.36018 2024-04-22T06:34:17.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1114 16 3599 3583 0.200992 0 - 0.36018 2024-04-22T06:34:18.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1115 16 3599 3583 0.200812 0 - 0.36018 2024-04-22T06:34:19.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1116 16 3599 3583 0.200632 0 - 0.36018 2024-04-22T06:34:20.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1117 16 3599 3583 0.200453 0 - 0.36018 2024-04-22T06:34:21.606 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1118 16 3599 3583 0.200273 0 - 0.36018 2024-04-22T06:34:22.607 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1119 16 3599 3583 0.200094 0 - 0.36018 2024-04-22T06:34:23.607 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:34:23.609696+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:34:23.607 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:34:23.607 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1120 16 3599 3583 0.199916 0 - 0.36018 2024-04-22T06:34:24.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1121 16 3599 3583 0.199737 0 - 0.36018 2024-04-22T06:34:25.607 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1122 16 3599 3583 0.199559 0 - 0.36018 2024-04-22T06:34:26.607 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1123 16 3599 3583 0.199382 0 - 0.36018 2024-04-22T06:34:27.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1124 16 3599 3583 0.199204 0 - 0.36018 2024-04-22T06:34:28.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1125 16 3599 3583 0.199027 0 - 0.36018 2024-04-22T06:34:29.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1126 16 3599 3583 0.19885 0 - 0.36018 2024-04-22T06:34:30.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1127 16 3599 3583 0.198674 0 - 0.36018 2024-04-22T06:34:31.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1128 16 3599 3583 0.198498 0 - 0.36018 2024-04-22T06:34:32.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1129 16 3599 3583 0.198322 0 - 0.36018 2024-04-22T06:34:33.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1130 16 3599 3583 0.198146 0 - 0.36018 2024-04-22T06:34:34.608 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1131 16 3599 3583 0.197971 0 - 0.36018 2024-04-22T06:34:35.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1132 16 3599 3583 0.197796 0 - 0.36018 2024-04-22T06:34:36.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1133 16 3599 3583 0.197622 0 - 0.36018 2024-04-22T06:34:37.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1134 16 3599 3583 0.197448 0 - 0.36018 2024-04-22T06:34:38.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1135 16 3599 3583 0.197274 0 - 0.36018 2024-04-22T06:34:39.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1136 16 3599 3583 0.1971 0 - 0.36018 2024-04-22T06:34:40.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1137 16 3599 3583 0.196927 0 - 0.36018 2024-04-22T06:34:41.609 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1138 16 3599 3583 0.196754 0 - 0.36018 2024-04-22T06:34:42.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1139 16 3599 3583 0.196581 0 - 0.36018 2024-04-22T06:34:43.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:34:43.612649+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:34:43.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:34:43.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1140 16 3599 3583 0.196408 0 - 0.36018 2024-04-22T06:34:44.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1141 16 3599 3583 0.196236 0 - 0.36018 2024-04-22T06:34:45.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1142 16 3599 3583 0.196064 0 - 0.36018 2024-04-22T06:34:46.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1143 16 3599 3583 0.195893 0 - 0.36018 2024-04-22T06:34:47.610 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1144 16 3599 3583 0.195722 0 - 0.36018 2024-04-22T06:34:48.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1145 16 3599 3583 0.195551 0 - 0.36018 2024-04-22T06:34:49.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1146 16 3599 3583 0.19538 0 - 0.36018 2024-04-22T06:34:50.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1147 16 3599 3583 0.19521 0 - 0.36018 2024-04-22T06:34:51.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1148 16 3599 3583 0.19504 0 - 0.36018 2024-04-22T06:34:52.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1149 16 3599 3583 0.19487 0 - 0.36018 2024-04-22T06:34:53.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1150 16 3599 3583 0.1947 0 - 0.36018 2024-04-22T06:34:54.612 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1151 16 3599 3583 0.194531 0 - 0.36018 2024-04-22T06:34:55.611 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1152 16 3599 3583 0.194362 0 - 0.36018 2024-04-22T06:34:56.612 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1153 16 3599 3583 0.194194 0 - 0.36018 2024-04-22T06:34:57.612 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1154 16 3599 3583 0.194026 0 - 0.36018 2024-04-22T06:34:58.612 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1155 16 3599 3583 0.193858 0 - 0.36018 2024-04-22T06:34:59.612 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1156 16 3599 3583 0.19369 0 - 0.36018 2024-04-22T06:35:00.612 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1157 16 3599 3583 0.193522 0 - 0.36018 2024-04-22T06:35:01.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1158 16 3599 3583 0.193355 0 - 0.36018 2024-04-22T06:35:02.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1159 16 3599 3583 0.193189 0 - 0.36018 2024-04-22T06:35:03.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:35:03.615601+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:35:03.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:35:03.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1160 16 3599 3583 0.193022 0 - 0.36018 2024-04-22T06:35:04.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1161 16 3599 3583 0.192856 0 - 0.36018 2024-04-22T06:35:05.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1162 16 3599 3583 0.19269 0 - 0.36018 2024-04-22T06:35:06.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1163 16 3599 3583 0.192524 0 - 0.36018 2024-04-22T06:35:07.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1164 16 3599 3583 0.192359 0 - 0.36018 2024-04-22T06:35:08.613 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1165 16 3599 3583 0.192194 0 - 0.36018 2024-04-22T06:35:09.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1166 16 3599 3583 0.192029 0 - 0.36018 2024-04-22T06:35:10.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1167 16 3599 3583 0.191864 0 - 0.36018 2024-04-22T06:35:11.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1168 16 3599 3583 0.1917 0 - 0.36018 2024-04-22T06:35:12.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1169 16 3599 3583 0.191536 0 - 0.36018 2024-04-22T06:35:13.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1170 16 3599 3583 0.191372 0 - 0.36018 2024-04-22T06:35:14.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1171 16 3599 3583 0.191209 0 - 0.36018 2024-04-22T06:35:15.614 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1172 16 3599 3583 0.191046 0 - 0.36018 2024-04-22T06:35:16.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1173 16 3599 3583 0.190883 0 - 0.36018 2024-04-22T06:35:17.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1174 16 3599 3583 0.19072 0 - 0.36018 2024-04-22T06:35:18.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1175 16 3599 3583 0.190558 0 - 0.36018 2024-04-22T06:35:19.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1176 16 3599 3583 0.190396 0 - 0.36018 2024-04-22T06:35:20.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1177 16 3599 3583 0.190234 0 - 0.36018 2024-04-22T06:35:21.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1178 16 3599 3583 0.190073 0 - 0.36018 2024-04-22T06:35:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:35:22.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:35:22.615 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1179 16 3599 3583 0.189911 0 - 0.36018 2024-04-22T06:35:23.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:35:23.618483+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:35:23.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:35:23.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1180 16 3599 3583 0.18975 0 - 0.36018 2024-04-22T06:35:24.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:35:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:35:24.248+0000 7f681a55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:35:24.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1181 16 3599 3583 0.18959 0 - 0.36018 2024-04-22T06:35:25.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1182 16 3599 3583 0.189429 0 - 0.36018 2024-04-22T06:35:26.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1183 16 3599 3583 0.189269 0 - 0.36018 2024-04-22T06:35:27.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1184 16 3599 3583 0.189109 0 - 0.36018 2024-04-22T06:35:28.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1185 16 3599 3583 0.18895 0 - 0.36018 2024-04-22T06:35:29.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1186 16 3599 3583 0.18879 0 - 0.36018 2024-04-22T06:35:30.616 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1187 16 3599 3583 0.188631 0 - 0.36018 2024-04-22T06:35:31.617 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1188 16 3599 3583 0.188473 0 - 0.36018 2024-04-22T06:35:32.617 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1189 16 3599 3583 0.188314 0 - 0.36018 2024-04-22T06:35:33.617 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1190 16 3599 3583 0.188156 0 - 0.36018 2024-04-22T06:35:34.617 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1191 16 3599 3583 0.187998 0 - 0.36018 2024-04-22T06:35:35.617 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1192 16 3599 3583 0.18784 0 - 0.36018 2024-04-22T06:35:36.617 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1193 16 3599 3583 0.187683 0 - 0.36018 2024-04-22T06:35:37.618 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1194 16 3599 3583 0.187526 0 - 0.36018 2024-04-22T06:35:38.618 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1195 16 3599 3583 0.187369 0 - 0.36018 2024-04-22T06:35:39.618 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1196 16 3599 3583 0.187212 0 - 0.36018 2024-04-22T06:35:40.618 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1197 16 3599 3583 0.187056 0 - 0.36018 2024-04-22T06:35:41.618 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1198 16 3599 3583 0.186899 0 - 0.36018 2024-04-22T06:35:42.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1199 16 3599 3583 0.186743 0 - 0.36018 2024-04-22T06:35:43.618 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:35:43.621432+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:35:43.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:35:43.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1200 16 3599 3583 0.186588 0 - 0.36018 2024-04-22T06:35:44.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1201 16 3599 3583 0.186433 0 - 0.36018 2024-04-22T06:35:45.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1202 16 3599 3583 0.186277 0 - 0.36018 2024-04-22T06:35:46.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1203 16 3599 3583 0.186123 0 - 0.36018 2024-04-22T06:35:47.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1204 16 3599 3583 0.185968 0 - 0.36018 2024-04-22T06:35:48.619 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1205 16 3599 3583 0.185814 0 - 0.36018 2024-04-22T06:35:49.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1206 16 3599 3583 0.18566 0 - 0.36018 2024-04-22T06:35:50.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1207 16 3599 3583 0.185506 0 - 0.36018 2024-04-22T06:35:51.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1208 16 3599 3583 0.185352 0 - 0.36018 2024-04-22T06:35:52.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1209 16 3599 3583 0.185199 0 - 0.36018 2024-04-22T06:35:53.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1210 16 3599 3583 0.185046 0 - 0.36018 2024-04-22T06:35:54.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1211 16 3599 3583 0.184893 0 - 0.36018 2024-04-22T06:35:55.620 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1212 16 3599 3583 0.18474 0 - 0.36018 2024-04-22T06:35:56.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1213 16 3599 3583 0.184588 0 - 0.36018 2024-04-22T06:35:57.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1214 16 3599 3583 0.184436 0 - 0.36018 2024-04-22T06:35:58.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1215 16 3599 3583 0.184284 0 - 0.36018 2024-04-22T06:35:59.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1216 16 3599 3583 0.184133 0 - 0.36018 2024-04-22T06:36:00.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1217 16 3599 3583 0.183981 0 - 0.36018 2024-04-22T06:36:01.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1218 16 3599 3583 0.18383 0 - 0.36018 2024-04-22T06:36:02.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1219 16 3599 3583 0.18368 0 - 0.36018 2024-04-22T06:36:03.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:36:03.624328+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:36:03.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:36:03.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1220 16 3599 3583 0.183529 0 - 0.36018 2024-04-22T06:36:04.621 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1221 16 3599 3583 0.183379 0 - 0.36018 2024-04-22T06:36:05.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1222 16 3599 3583 0.183229 0 - 0.36018 2024-04-22T06:36:06.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1223 16 3599 3583 0.183079 0 - 0.36018 2024-04-22T06:36:07.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1224 16 3599 3583 0.182929 0 - 0.36018 2024-04-22T06:36:08.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1225 16 3599 3583 0.18278 0 - 0.36018 2024-04-22T06:36:09.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1226 16 3599 3583 0.182631 0 - 0.36018 2024-04-22T06:36:10.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1227 16 3599 3583 0.182482 0 - 0.36018 2024-04-22T06:36:11.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1228 16 3599 3583 0.182333 0 - 0.36018 2024-04-22T06:36:12.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1229 16 3599 3583 0.182185 0 - 0.36018 2024-04-22T06:36:13.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1230 16 3599 3583 0.182037 0 - 0.36018 2024-04-22T06:36:14.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1231 16 3599 3583 0.181889 0 - 0.36018 2024-04-22T06:36:15.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1232 16 3599 3583 0.181741 0 - 0.36018 2024-04-22T06:36:16.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1233 16 3599 3583 0.181594 0 - 0.36018 2024-04-22T06:36:17.623 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1234 16 3599 3583 0.181447 0 - 0.36018 2024-04-22T06:36:18.624 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1235 16 3599 3583 0.1813 0 - 0.36018 2024-04-22T06:36:19.624 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1236 16 3599 3583 0.181153 0 - 0.36018 2024-04-22T06:36:20.624 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1237 16 3599 3583 0.181007 0 - 0.36018 2024-04-22T06:36:21.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1238 16 3599 3583 0.180861 0 - 0.36018 2024-04-22T06:36:22.624 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1239 16 3599 3583 0.180715 0 - 0.36018 2024-04-22T06:36:23.624 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:36:23.627338+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:36:23.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:36:23.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1240 16 3599 3583 0.180569 0 - 0.36018 2024-04-22T06:36:24.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1241 16 3599 3583 0.180423 0 - 0.36018 2024-04-22T06:36:25.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1242 16 3599 3583 0.180278 0 - 0.36018 2024-04-22T06:36:26.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1243 16 3599 3583 0.180133 0 - 0.36018 2024-04-22T06:36:27.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1244 16 3599 3583 0.179988 0 - 0.36018 2024-04-22T06:36:28.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1245 16 3599 3583 0.179844 0 - 0.36018 2024-04-22T06:36:29.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1246 16 3599 3583 0.179699 0 - 0.36018 2024-04-22T06:36:30.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1247 16 3599 3583 0.179555 0 - 0.36018 2024-04-22T06:36:31.625 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1248 16 3599 3583 0.179411 0 - 0.36018 2024-04-22T06:36:32.626 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1249 16 3599 3583 0.179268 0 - 0.36018 2024-04-22T06:36:33.626 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1250 16 3599 3583 0.179124 0 - 0.36018 2024-04-22T06:36:34.626 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1251 16 3599 3583 0.178981 0 - 0.36018 2024-04-22T06:36:35.626 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1252 16 3599 3583 0.178838 0 - 0.36018 2024-04-22T06:36:36.626 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1253 16 3599 3583 0.178695 0 - 0.36018 2024-04-22T06:36:37.626 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1254 16 3599 3583 0.178553 0 - 0.36018 2024-04-22T06:36:38.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1255 16 3599 3583 0.178411 0 - 0.36018 2024-04-22T06:36:39.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1256 16 3599 3583 0.178269 0 - 0.36018 2024-04-22T06:36:40.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1257 16 3599 3583 0.178127 0 - 0.36018 2024-04-22T06:36:41.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1258 16 3599 3583 0.177985 0 - 0.36018 2024-04-22T06:36:42.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1259 16 3599 3583 0.177844 0 - 0.36018 2024-04-22T06:36:43.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:36:43.630247+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:36:43.627 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:36:43.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1260 16 3599 3583 0.177703 0 - 0.36018 2024-04-22T06:36:44.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1261 16 3599 3583 0.177562 0 - 0.36018 2024-04-22T06:36:45.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1262 16 3599 3583 0.177421 0 - 0.36018 2024-04-22T06:36:46.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1263 16 3599 3583 0.177281 0 - 0.36018 2024-04-22T06:36:47.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1264 16 3599 3583 0.17714 0 - 0.36018 2024-04-22T06:36:48.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1265 16 3599 3583 0.177 0 - 0.36018 2024-04-22T06:36:49.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1266 16 3599 3583 0.176861 0 - 0.36018 2024-04-22T06:36:50.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1267 16 3599 3583 0.176721 0 - 0.36018 2024-04-22T06:36:51.628 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1268 16 3599 3583 0.176582 0 - 0.36018 2024-04-22T06:36:52.629 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1269 16 3599 3583 0.176442 0 - 0.36018 2024-04-22T06:36:53.629 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1270 16 3599 3583 0.176303 0 - 0.36018 2024-04-22T06:36:54.629 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1271 16 3599 3583 0.176165 0 - 0.36018 2024-04-22T06:36:55.629 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1272 16 3599 3583 0.176026 0 - 0.36018 2024-04-22T06:36:56.629 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1273 16 3599 3583 0.175888 0 - 0.36018 2024-04-22T06:36:57.629 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1274 16 3599 3583 0.17575 0 - 0.36018 2024-04-22T06:36:58.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1275 16 3599 3583 0.175612 0 - 0.36018 2024-04-22T06:36:59.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1276 16 3599 3583 0.175474 0 - 0.36018 2024-04-22T06:37:00.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1277 16 3599 3583 0.175337 0 - 0.36018 2024-04-22T06:37:01.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1278 16 3599 3583 0.1752 0 - 0.36018 2024-04-22T06:37:02.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1279 16 3599 3583 0.175063 0 - 0.36018 2024-04-22T06:37:03.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:37:03.633127+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:37:03.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:37:03.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1280 16 3599 3583 0.174926 0 - 0.36018 2024-04-22T06:37:04.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1281 16 3599 3583 0.17479 0 - 0.36018 2024-04-22T06:37:05.630 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1282 16 3599 3583 0.174653 0 - 0.36018 2024-04-22T06:37:06.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1283 16 3599 3583 0.174517 0 - 0.36018 2024-04-22T06:37:07.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1284 16 3599 3583 0.174381 0 - 0.36018 2024-04-22T06:37:08.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1285 16 3599 3583 0.174245 0 - 0.36018 2024-04-22T06:37:09.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1286 16 3599 3583 0.17411 0 - 0.36018 2024-04-22T06:37:10.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1287 16 3599 3583 0.173975 0 - 0.36018 2024-04-22T06:37:11.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1288 16 3599 3583 0.17384 0 - 0.36018 2024-04-22T06:37:12.631 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1289 16 3599 3583 0.173705 0 - 0.36018 2024-04-22T06:37:13.632 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1290 16 3599 3583 0.17357 0 - 0.36018 2024-04-22T06:37:14.632 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1291 16 3599 3583 0.173436 0 - 0.36018 2024-04-22T06:37:15.632 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1292 16 3599 3583 0.173301 0 - 0.36018 2024-04-22T06:37:16.632 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1293 16 3599 3583 0.173167 0 - 0.36018 2024-04-22T06:37:17.632 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1294 16 3599 3583 0.173034 0 - 0.36018 2024-04-22T06:37:18.632 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1295 16 3599 3583 0.1729 0 - 0.36018 2024-04-22T06:37:19.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1296 16 3599 3583 0.172767 0 - 0.36018 2024-04-22T06:37:20.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1297 16 3599 3583 0.172633 0 - 0.36018 2024-04-22T06:37:21.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1298 16 3599 3583 0.1725 0 - 0.36018 2024-04-22T06:37:22.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1299 16 3599 3583 0.172368 0 - 0.36018 2024-04-22T06:37:23.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:37:23.636061+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:37:23.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:37:23.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1300 16 3599 3583 0.172235 0 - 0.36018 2024-04-22T06:37:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:37:24.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:37:24.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1301 16 3599 3583 0.172103 0 - 0.36018 2024-04-22T06:37:25.633 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1302 16 3599 3583 0.17197 0 - 0.36018 2024-04-22T06:37:26.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:37:26.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:37:26.284+0000 7f500031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:37:26.634 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1303 16 3599 3583 0.171838 0 - 0.36018 2024-04-22T06:37:27.634 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1304 16 3599 3583 0.171707 0 - 0.36018 2024-04-22T06:37:28.634 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1305 16 3599 3583 0.171575 0 - 0.36018 2024-04-22T06:37:29.634 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1306 16 3599 3583 0.171444 0 - 0.36018 2024-04-22T06:37:30.634 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1307 16 3599 3583 0.171312 0 - 0.36018 2024-04-22T06:37:31.634 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1308 16 3599 3583 0.171182 0 - 0.36018 2024-04-22T06:37:32.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1309 16 3599 3583 0.171051 0 - 0.36018 2024-04-22T06:37:33.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1310 16 3599 3583 0.17092 0 - 0.36018 2024-04-22T06:37:34.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1311 16 3599 3583 0.17079 0 - 0.36018 2024-04-22T06:37:35.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1312 16 3599 3583 0.17066 0 - 0.36018 2024-04-22T06:37:36.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1313 16 3599 3583 0.17053 0 - 0.36018 2024-04-22T06:37:37.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1314 16 3599 3583 0.1704 0 - 0.36018 2024-04-22T06:37:38.635 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1315 16 3599 3583 0.17027 0 - 0.36018 2024-04-22T06:37:39.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1316 16 3599 3583 0.170141 0 - 0.36018 2024-04-22T06:37:40.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1317 16 3599 3583 0.170012 0 - 0.36018 2024-04-22T06:37:41.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1318 16 3599 3583 0.169883 0 - 0.36018 2024-04-22T06:37:42.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1319 16 3599 3583 0.169754 0 - 0.36018 2024-04-22T06:37:43.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:37:43.638913+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:37:43.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:37:43.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1320 16 3599 3583 0.169625 0 - 0.36018 2024-04-22T06:37:44.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1321 16 3599 3583 0.169497 0 - 0.36018 2024-04-22T06:37:45.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1322 16 3599 3583 0.169369 0 - 0.36018 2024-04-22T06:37:46.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1323 16 3599 3583 0.169241 0 - 0.36018 2024-04-22T06:37:47.636 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1324 16 3599 3583 0.169113 0 - 0.36018 2024-04-22T06:37:48.637 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1325 16 3599 3583 0.168985 0 - 0.36018 2024-04-22T06:37:49.637 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1326 16 3599 3583 0.168858 0 - 0.36018 2024-04-22T06:37:50.637 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1327 16 3599 3583 0.168731 0 - 0.36018 2024-04-22T06:37:51.637 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1328 16 3599 3583 0.168603 0 - 0.36018 2024-04-22T06:37:52.637 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1329 16 3599 3583 0.168477 0 - 0.36018 2024-04-22T06:37:53.637 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1330 16 3599 3583 0.16835 0 - 0.36018 2024-04-22T06:37:54.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1331 16 3599 3583 0.168223 0 - 0.36018 2024-04-22T06:37:55.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1332 16 3599 3583 0.168097 0 - 0.36018 2024-04-22T06:37:56.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1333 16 3599 3583 0.167971 0 - 0.36018 2024-04-22T06:37:57.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1334 16 3599 3583 0.167845 0 - 0.36018 2024-04-22T06:37:58.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1335 16 3599 3583 0.167719 0 - 0.36018 2024-04-22T06:37:59.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1336 16 3599 3583 0.167594 0 - 0.36018 2024-04-22T06:38:00.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1337 16 3599 3583 0.167469 0 - 0.36018 2024-04-22T06:38:01.638 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1338 16 3599 3583 0.167343 0 - 0.36018 2024-04-22T06:38:02.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1339 16 3599 3583 0.167218 0 - 0.36018 2024-04-22T06:38:03.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:38:03.641876+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:38:03.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:38:03.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1340 16 3599 3583 0.167094 0 - 0.36018 2024-04-22T06:38:04.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1341 16 3599 3583 0.166969 0 - 0.36018 2024-04-22T06:38:05.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1342 16 3599 3583 0.166845 0 - 0.36018 2024-04-22T06:38:06.639 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1343 16 3599 3583 0.16672 0 - 0.36018 2024-04-22T06:38:07.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1344 16 3599 3583 0.166596 0 - 0.36018 2024-04-22T06:38:08.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1345 16 3599 3583 0.166472 0 - 0.36018 2024-04-22T06:38:09.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1346 16 3599 3583 0.166349 0 - 0.36018 2024-04-22T06:38:10.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1347 16 3599 3583 0.166225 0 - 0.36018 2024-04-22T06:38:11.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1348 16 3599 3583 0.166102 0 - 0.36018 2024-04-22T06:38:12.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1349 16 3599 3583 0.165979 0 - 0.36018 2024-04-22T06:38:13.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1350 16 3599 3583 0.165856 0 - 0.36018 2024-04-22T06:38:14.640 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1351 16 3599 3583 0.165733 0 - 0.36018 2024-04-22T06:38:15.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1352 16 3599 3583 0.165611 0 - 0.36018 2024-04-22T06:38:16.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1353 16 3599 3583 0.165488 0 - 0.36018 2024-04-22T06:38:17.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1354 16 3599 3583 0.165366 0 - 0.36018 2024-04-22T06:38:18.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1355 16 3599 3583 0.165244 0 - 0.36018 2024-04-22T06:38:19.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1356 16 3599 3583 0.165122 0 - 0.36018 2024-04-22T06:38:20.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1357 16 3599 3583 0.165 0 - 0.36018 2024-04-22T06:38:21.641 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1358 16 3599 3583 0.164879 0 - 0.36018 2024-04-22T06:38:22.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1359 16 3599 3583 0.164757 0 - 0.36018 2024-04-22T06:38:23.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:38:23.644652+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:38:23.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:38:23.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1360 16 3599 3583 0.164636 0 - 0.36018 2024-04-22T06:38:24.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1361 16 3599 3583 0.164515 0 - 0.36018 2024-04-22T06:38:25.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1362 16 3599 3583 0.164395 0 - 0.36018 2024-04-22T06:38:26.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1363 16 3599 3583 0.164274 0 - 0.36018 2024-04-22T06:38:27.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1364 16 3599 3583 0.164154 0 - 0.36018 2024-04-22T06:38:28.642 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1365 16 3599 3583 0.164033 0 - 0.36018 2024-04-22T06:38:29.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1366 16 3599 3583 0.163913 0 - 0.36018 2024-04-22T06:38:30.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1367 16 3599 3583 0.163793 0 - 0.36018 2024-04-22T06:38:31.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1368 16 3599 3583 0.163674 0 - 0.36018 2024-04-22T06:38:32.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1369 16 3599 3583 0.163554 0 - 0.36018 2024-04-22T06:38:33.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1370 16 3599 3583 0.163435 0 - 0.36018 2024-04-22T06:38:34.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1371 16 3599 3583 0.163315 0 - 0.36018 2024-04-22T06:38:35.644 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1372 16 3599 3583 0.163196 0 - 0.36018 2024-04-22T06:38:36.643 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1373 16 3599 3583 0.163077 0 - 0.36018 2024-04-22T06:38:37.644 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1374 16 3599 3583 0.162959 0 - 0.36018 2024-04-22T06:38:38.644 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1375 16 3599 3583 0.16284 0 - 0.36018 2024-04-22T06:38:39.644 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1376 16 3599 3583 0.162722 0 - 0.36018 2024-04-22T06:38:40.644 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1377 16 3599 3583 0.162604 0 - 0.36018 2024-04-22T06:38:41.644 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1378 16 3599 3583 0.162486 0 - 0.36018 2024-04-22T06:38:42.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1379 16 3599 3583 0.162368 0 - 0.36018 2024-04-22T06:38:43.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:38:43.647559+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:38:43.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:38:43.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1380 16 3599 3583 0.16225 0 - 0.36018 2024-04-22T06:38:44.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1381 16 3599 3583 0.162133 0 - 0.36018 2024-04-22T06:38:45.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1382 16 3599 3583 0.162015 0 - 0.36018 2024-04-22T06:38:46.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1383 16 3599 3583 0.161898 0 - 0.36018 2024-04-22T06:38:47.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1384 16 3599 3583 0.161781 0 - 0.36018 2024-04-22T06:38:48.645 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1385 16 3599 3583 0.161665 0 - 0.36018 2024-04-22T06:38:49.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1386 16 3599 3583 0.161548 0 - 0.36018 2024-04-22T06:38:50.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1387 16 3599 3583 0.161431 0 - 0.36018 2024-04-22T06:38:51.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1388 16 3599 3583 0.161315 0 - 0.36018 2024-04-22T06:38:52.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1389 16 3599 3583 0.161199 0 - 0.36018 2024-04-22T06:38:53.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1390 16 3599 3583 0.161083 0 - 0.36018 2024-04-22T06:38:54.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1391 16 3599 3583 0.160967 0 - 0.36018 2024-04-22T06:38:55.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1392 16 3599 3583 0.160852 0 - 0.36018 2024-04-22T06:38:56.646 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1393 16 3599 3583 0.160736 0 - 0.36018 2024-04-22T06:38:57.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1394 16 3599 3583 0.160621 0 - 0.36018 2024-04-22T06:38:58.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1395 16 3599 3583 0.160506 0 - 0.36018 2024-04-22T06:38:59.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1396 16 3599 3583 0.160391 0 - 0.36018 2024-04-22T06:39:00.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1397 16 3599 3583 0.160276 0 - 0.36018 2024-04-22T06:39:01.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1398 16 3599 3583 0.160161 0 - 0.36018 2024-04-22T06:39:02.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1399 16 3599 3583 0.160047 0 - 0.36018 2024-04-22T06:39:03.647 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:39:03.650364+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:39:03.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:39:03.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1400 16 3599 3583 0.159932 0 - 0.36018 2024-04-22T06:39:04.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1401 16 3599 3583 0.159818 0 - 0.36018 2024-04-22T06:39:05.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1402 16 3599 3583 0.159704 0 - 0.36018 2024-04-22T06:39:06.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1403 16 3599 3583 0.15959 0 - 0.36018 2024-04-22T06:39:07.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1404 16 3599 3583 0.159477 0 - 0.36018 2024-04-22T06:39:08.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1405 16 3599 3583 0.159363 0 - 0.36018 2024-04-22T06:39:09.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1406 16 3599 3583 0.15925 0 - 0.36018 2024-04-22T06:39:10.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1407 16 3599 3583 0.159137 0 - 0.36018 2024-04-22T06:39:11.648 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1408 16 3599 3583 0.159024 0 - 0.36018 2024-04-22T06:39:12.649 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1409 16 3599 3583 0.158911 0 - 0.36018 2024-04-22T06:39:13.649 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1410 16 3599 3583 0.158798 0 - 0.36018 2024-04-22T06:39:14.649 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1411 16 3599 3583 0.158686 0 - 0.36018 2024-04-22T06:39:15.649 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1412 16 3599 3583 0.158573 0 - 0.36018 2024-04-22T06:39:16.649 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1413 16 3599 3583 0.158461 0 - 0.36018 2024-04-22T06:39:17.649 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1414 16 3599 3583 0.158349 0 - 0.36018 2024-04-22T06:39:18.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1415 16 3599 3583 0.158237 0 - 0.36018 2024-04-22T06:39:19.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1416 16 3599 3583 0.158125 0 - 0.36018 2024-04-22T06:39:20.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1417 16 3599 3583 0.158014 0 - 0.36018 2024-04-22T06:39:21.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1418 16 3599 3583 0.157902 0 - 0.36018 2024-04-22T06:39:22.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1419 16 3599 3583 0.157791 0 - 0.36018 2024-04-22T06:39:23.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:39:23.653193+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:39:23.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:39:23.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1420 16 3599 3583 0.15768 0 - 0.36018 2024-04-22T06:39:24.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1421 16 3599 3583 0.157569 0 - 0.36018 2024-04-22T06:39:25.650 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1422 16 3599 3583 0.157458 0 - 0.36018 2024-04-22T06:39:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:39:26.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:39:26.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1423 16 3599 3583 0.157347 0 - 0.36018 2024-04-22T06:39:27.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1424 16 3599 3583 0.157237 0 - 0.36018 2024-04-22T06:39:28.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:39:28.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:39:28.320+0000 7f81df06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:39:28.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1425 16 3599 3583 0.157127 0 - 0.36018 2024-04-22T06:39:29.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1426 16 3599 3583 0.157016 0 - 0.36018 2024-04-22T06:39:30.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1427 16 3599 3583 0.156906 0 - 0.36018 2024-04-22T06:39:31.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1428 16 3599 3583 0.156797 0 - 0.36018 2024-04-22T06:39:32.651 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1429 16 3599 3583 0.156687 0 - 0.36018 2024-04-22T06:39:33.652 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1430 16 3599 3583 0.156577 0 - 0.36018 2024-04-22T06:39:34.652 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1431 16 3599 3583 0.156468 0 - 0.36018 2024-04-22T06:39:35.652 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1432 16 3599 3583 0.156359 0 - 0.36018 2024-04-22T06:39:36.652 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1433 16 3599 3583 0.156249 0 - 0.36018 2024-04-22T06:39:37.652 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1434 16 3599 3583 0.15614 0 - 0.36018 2024-04-22T06:39:38.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1435 16 3599 3583 0.156032 0 - 0.36018 2024-04-22T06:39:39.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1436 16 3599 3583 0.155923 0 - 0.36018 2024-04-22T06:39:40.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1437 16 3599 3583 0.155814 0 - 0.36018 2024-04-22T06:39:41.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1438 16 3599 3583 0.155706 0 - 0.36018 2024-04-22T06:39:42.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1439 16 3599 3583 0.155598 0 - 0.36018 2024-04-22T06:39:43.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:39:43.656081+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:39:43.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:39:43.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1440 16 3599 3583 0.15549 0 - 0.36018 2024-04-22T06:39:44.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1441 16 3599 3583 0.155382 0 - 0.36018 2024-04-22T06:39:45.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1442 16 3599 3583 0.155274 0 - 0.36018 2024-04-22T06:39:46.653 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1443 16 3599 3583 0.155167 0 - 0.36018 2024-04-22T06:39:47.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1444 16 3599 3583 0.155059 0 - 0.36018 2024-04-22T06:39:48.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1445 16 3599 3583 0.154952 0 - 0.36018 2024-04-22T06:39:49.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1446 16 3599 3583 0.154845 0 - 0.36018 2024-04-22T06:39:50.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1447 16 3599 3583 0.154738 0 - 0.36018 2024-04-22T06:39:51.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1448 16 3599 3583 0.154631 0 - 0.36018 2024-04-22T06:39:52.654 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1449 16 3599 3583 0.154524 0 - 0.36018 2024-04-22T06:39:53.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1450 16 3599 3583 0.154418 0 - 0.36018 2024-04-22T06:39:54.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1451 16 3599 3583 0.154311 0 - 0.36018 2024-04-22T06:39:55.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1452 16 3599 3583 0.154205 0 - 0.36018 2024-04-22T06:39:56.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1453 16 3599 3583 0.154099 0 - 0.36018 2024-04-22T06:39:57.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1454 16 3599 3583 0.153993 0 - 0.36018 2024-04-22T06:39:58.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1455 16 3599 3583 0.153887 0 - 0.36018 2024-04-22T06:39:59.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1456 16 3599 3583 0.153781 0 - 0.36018 2024-04-22T06:40:00.655 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1457 16 3599 3583 0.153676 0 - 0.36018 2024-04-22T06:40:01.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1458 16 3599 3583 0.15357 0 - 0.36018 2024-04-22T06:40:02.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1459 16 3599 3583 0.153465 0 - 0.36018 2024-04-22T06:40:03.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:40:03.658936+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:40:03.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:40:03.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1460 16 3599 3583 0.15336 0 - 0.36018 2024-04-22T06:40:04.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1461 16 3599 3583 0.153255 0 - 0.36018 2024-04-22T06:40:05.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1462 16 3599 3583 0.15315 0 - 0.36018 2024-04-22T06:40:06.656 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1463 16 3599 3583 0.153045 0 - 0.36018 2024-04-22T06:40:07.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1464 16 3599 3583 0.152941 0 - 0.36018 2024-04-22T06:40:08.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1465 16 3599 3583 0.152836 0 - 0.36018 2024-04-22T06:40:09.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1466 16 3599 3583 0.152732 0 - 0.36018 2024-04-22T06:40:10.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1467 16 3599 3583 0.152628 0 - 0.36018 2024-04-22T06:40:11.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1468 16 3599 3583 0.152524 0 - 0.36018 2024-04-22T06:40:12.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1469 16 3599 3583 0.15242 0 - 0.36018 2024-04-22T06:40:13.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1470 16 3599 3583 0.152317 0 - 0.36018 2024-04-22T06:40:14.657 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1471 16 3599 3583 0.152213 0 - 0.36018 2024-04-22T06:40:15.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1472 16 3599 3583 0.15211 0 - 0.36018 2024-04-22T06:40:16.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1473 16 3599 3583 0.152006 0 - 0.36018 2024-04-22T06:40:17.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1474 16 3599 3583 0.151903 0 - 0.36018 2024-04-22T06:40:18.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1475 16 3599 3583 0.1518 0 - 0.36018 2024-04-22T06:40:19.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1476 16 3599 3583 0.151697 0 - 0.36018 2024-04-22T06:40:20.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1477 16 3599 3583 0.151595 0 - 0.36018 2024-04-22T06:40:21.658 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1478 16 3599 3583 0.151492 0 - 0.36018 2024-04-22T06:40:22.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1479 16 3599 3583 0.15139 0 - 0.36018 2024-04-22T06:40:23.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:40:23.661800+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:40:23.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:40:23.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1480 16 3599 3583 0.151287 0 - 0.36018 2024-04-22T06:40:24.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1481 16 3599 3583 0.151185 0 - 0.36018 2024-04-22T06:40:25.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1482 16 3599 3583 0.151083 0 - 0.36018 2024-04-22T06:40:26.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1483 16 3599 3583 0.150981 0 - 0.36018 2024-04-22T06:40:27.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1484 16 3599 3583 0.15088 0 - 0.36018 2024-04-22T06:40:28.659 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1485 16 3599 3583 0.150778 0 - 0.36018 2024-04-22T06:40:29.660 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1486 16 3599 3583 0.150677 0 - 0.36018 2024-04-22T06:40:30.660 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1487 16 3599 3583 0.150575 0 - 0.36018 2024-04-22T06:40:31.660 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1488 16 3599 3583 0.150474 0 - 0.36018 2024-04-22T06:40:32.660 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1489 16 3599 3583 0.150373 0 - 0.36018 2024-04-22T06:40:33.660 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1490 16 3599 3583 0.150272 0 - 0.36018 2024-04-22T06:40:34.661 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1491 16 3599 3583 0.150171 0 - 0.36018 2024-04-22T06:40:35.661 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1492 16 3599 3583 0.150071 0 - 0.36018 2024-04-22T06:40:36.661 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1493 16 3599 3583 0.14997 0 - 0.36018 2024-04-22T06:40:37.661 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1494 16 3599 3583 0.14987 0 - 0.36018 2024-04-22T06:40:38.661 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1495 16 3599 3583 0.149769 0 - 0.36018 2024-04-22T06:40:39.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1496 16 3599 3583 0.149669 0 - 0.36018 2024-04-22T06:40:40.661 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1497 16 3599 3583 0.149569 0 - 0.36018 2024-04-22T06:40:41.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1498 16 3599 3583 0.14947 0 - 0.36018 2024-04-22T06:40:42.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1499 16 3599 3583 0.14937 0 - 0.36018 2024-04-22T06:40:43.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:40:43.664812+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:40:43.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:40:43.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1500 16 3599 3583 0.14927 0 - 0.36018 2024-04-22T06:40:44.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1501 16 3599 3583 0.149171 0 - 0.36018 2024-04-22T06:40:45.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1502 16 3599 3583 0.149071 0 - 0.36018 2024-04-22T06:40:46.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1503 16 3599 3583 0.148972 0 - 0.36018 2024-04-22T06:40:47.662 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1504 16 3599 3583 0.148873 0 - 0.36018 2024-04-22T06:40:48.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1505 16 3599 3583 0.148774 0 - 0.36018 2024-04-22T06:40:49.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1506 16 3599 3583 0.148676 0 - 0.36018 2024-04-22T06:40:50.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1507 16 3599 3583 0.148577 0 - 0.36018 2024-04-22T06:40:51.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1508 16 3599 3583 0.148478 0 - 0.36018 2024-04-22T06:40:52.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1509 16 3599 3583 0.14838 0 - 0.36018 2024-04-22T06:40:53.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1510 16 3599 3583 0.148282 0 - 0.36018 2024-04-22T06:40:54.663 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1511 16 3599 3583 0.148184 0 - 0.36018 2024-04-22T06:40:55.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1512 16 3599 3583 0.148086 0 - 0.36018 2024-04-22T06:40:56.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1513 16 3599 3583 0.147988 0 - 0.36018 2024-04-22T06:40:57.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1514 16 3599 3583 0.14789 0 - 0.36018 2024-04-22T06:40:58.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1515 16 3599 3583 0.147792 0 - 0.36018 2024-04-22T06:40:59.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1516 16 3599 3583 0.147695 0 - 0.36018 2024-04-22T06:41:00.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1517 16 3599 3583 0.147597 0 - 0.36018 2024-04-22T06:41:01.664 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1518 16 3599 3583 0.1475 0 - 0.36018 2024-04-22T06:41:02.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1519 16 3599 3583 0.147403 0 - 0.36018 2024-04-22T06:41:03.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:41:03.667854+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:41:03.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:41:03.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1520 16 3599 3583 0.147306 0 - 0.36018 2024-04-22T06:41:04.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1521 16 3599 3583 0.147209 0 - 0.36018 2024-04-22T06:41:05.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1522 16 3599 3583 0.147113 0 - 0.36018 2024-04-22T06:41:06.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1523 16 3599 3583 0.147016 0 - 0.36018 2024-04-22T06:41:07.665 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1524 16 3599 3583 0.14692 0 - 0.36018 2024-04-22T06:41:08.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1525 16 3599 3583 0.146823 0 - 0.36018 2024-04-22T06:41:09.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1526 16 3599 3583 0.146727 0 - 0.36018 2024-04-22T06:41:10.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1527 16 3599 3583 0.146631 0 - 0.36018 2024-04-22T06:41:11.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1528 16 3599 3583 0.146535 0 - 0.36018 2024-04-22T06:41:12.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1529 16 3599 3583 0.146439 0 - 0.36018 2024-04-22T06:41:13.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1530 16 3599 3583 0.146343 0 - 0.36018 2024-04-22T06:41:14.666 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1531 16 3599 3583 0.146248 0 - 0.36018 2024-04-22T06:41:15.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1532 16 3599 3583 0.146152 0 - 0.36018 2024-04-22T06:41:16.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1533 16 3599 3583 0.146057 0 - 0.36018 2024-04-22T06:41:17.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1534 16 3599 3583 0.145962 0 - 0.36018 2024-04-22T06:41:18.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1535 16 3599 3583 0.145867 0 - 0.36018 2024-04-22T06:41:19.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1536 16 3599 3583 0.145772 0 - 0.36018 2024-04-22T06:41:20.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1537 16 3599 3583 0.145677 0 - 0.36018 2024-04-22T06:41:21.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1538 16 3599 3583 0.145582 0 - 0.36018 2024-04-22T06:41:22.667 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1539 16 3599 3583 0.145488 0 - 0.36018 2024-04-22T06:41:23.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:41:23.670630+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:41:23.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:41:23.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1540 16 3599 3583 0.145393 0 - 0.36018 2024-04-22T06:41:24.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1541 16 3599 3583 0.145299 0 - 0.36018 2024-04-22T06:41:25.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1542 16 3599 3583 0.145205 0 - 0.36018 2024-04-22T06:41:26.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1543 16 3599 3583 0.14511 0 - 0.36018 2024-04-22T06:41:27.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1544 16 3599 3583 0.145016 0 - 0.36018 2024-04-22T06:41:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:41:28.234 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:41:28.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1545 16 3599 3583 0.144923 0 - 0.36018 2024-04-22T06:41:29.668 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1546 16 3599 3583 0.144829 0 - 0.36018 2024-04-22T06:41:30.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:41:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:41:30.357+0000 7f33f786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:41:30.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1547 16 3599 3583 0.144735 0 - 0.36018 2024-04-22T06:41:31.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1548 16 3599 3583 0.144642 0 - 0.36018 2024-04-22T06:41:32.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1549 16 3599 3583 0.144548 0 - 0.36018 2024-04-22T06:41:33.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1550 16 3599 3583 0.144455 0 - 0.36018 2024-04-22T06:41:34.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1551 16 3599 3583 0.144362 0 - 0.36018 2024-04-22T06:41:35.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1552 16 3599 3583 0.144269 0 - 0.36018 2024-04-22T06:41:36.669 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1553 16 3599 3583 0.144176 0 - 0.36018 2024-04-22T06:41:37.670 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1554 16 3599 3583 0.144083 0 - 0.36018 2024-04-22T06:41:38.670 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1555 16 3599 3583 0.143991 0 - 0.36018 2024-04-22T06:41:39.670 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1556 16 3599 3583 0.143898 0 - 0.36018 2024-04-22T06:41:40.670 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1557 16 3599 3583 0.143806 0 - 0.36018 2024-04-22T06:41:41.670 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1558 16 3599 3583 0.143713 0 - 0.36018 2024-04-22T06:41:42.670 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1559 16 3599 3583 0.143621 0 - 0.36018 2024-04-22T06:41:43.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:41:43.673591+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:41:43.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:41:43.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1560 16 3599 3583 0.143529 0 - 0.36018 2024-04-22T06:41:44.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1561 16 3599 3583 0.143437 0 - 0.36018 2024-04-22T06:41:45.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1562 16 3599 3583 0.143345 0 - 0.36018 2024-04-22T06:41:46.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1563 16 3599 3583 0.143254 0 - 0.36018 2024-04-22T06:41:47.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1564 16 3599 3583 0.143162 0 - 0.36018 2024-04-22T06:41:48.671 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1565 16 3599 3583 0.143071 0 - 0.36018 2024-04-22T06:41:49.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1566 16 3599 3583 0.142979 0 - 0.36018 2024-04-22T06:41:50.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1567 16 3599 3583 0.142888 0 - 0.36018 2024-04-22T06:41:51.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1568 16 3599 3583 0.142797 0 - 0.36018 2024-04-22T06:41:52.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1569 16 3599 3583 0.142706 0 - 0.36018 2024-04-22T06:41:53.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1570 16 3599 3583 0.142615 0 - 0.36018 2024-04-22T06:41:54.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1571 16 3599 3583 0.142524 0 - 0.36018 2024-04-22T06:41:55.672 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1572 16 3599 3583 0.142433 0 - 0.36018 2024-04-22T06:41:56.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1573 16 3599 3583 0.142343 0 - 0.36018 2024-04-22T06:41:57.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1574 16 3599 3583 0.142252 0 - 0.36018 2024-04-22T06:41:58.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1575 16 3599 3583 0.142162 0 - 0.36018 2024-04-22T06:41:59.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1576 16 3599 3583 0.142072 0 - 0.36018 2024-04-22T06:42:00.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1577 16 3599 3583 0.141982 0 - 0.36018 2024-04-22T06:42:01.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1578 16 3599 3583 0.141892 0 - 0.36018 2024-04-22T06:42:02.673 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1579 16 3599 3583 0.141802 0 - 0.36018 2024-04-22T06:42:03.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:42:03.676618+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:42:03.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:42:03.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1580 16 3599 3583 0.141712 0 - 0.36018 2024-04-22T06:42:04.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1581 16 3599 3583 0.141623 0 - 0.36018 2024-04-22T06:42:05.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1582 16 3599 3583 0.141533 0 - 0.36018 2024-04-22T06:42:06.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1583 16 3599 3583 0.141444 0 - 0.36018 2024-04-22T06:42:07.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1584 16 3599 3583 0.141354 0 - 0.36018 2024-04-22T06:42:08.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1585 16 3599 3583 0.141265 0 - 0.36018 2024-04-22T06:42:09.674 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1586 16 3599 3583 0.141176 0 - 0.36018 2024-04-22T06:42:10.675 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1587 16 3599 3583 0.141087 0 - 0.36018 2024-04-22T06:42:11.675 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1588 16 3599 3583 0.140998 0 - 0.36018 2024-04-22T06:42:12.675 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1589 16 3599 3583 0.14091 0 - 0.36018 2024-04-22T06:42:13.675 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1590 16 3599 3583 0.140821 0 - 0.36018 2024-04-22T06:42:14.675 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1591 16 3599 3583 0.140732 0 - 0.36018 2024-04-22T06:42:15.675 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1592 16 3599 3583 0.140644 0 - 0.36018 2024-04-22T06:42:16.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1593 16 3599 3583 0.140556 0 - 0.36018 2024-04-22T06:42:17.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1594 16 3599 3583 0.140468 0 - 0.36018 2024-04-22T06:42:18.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1595 16 3599 3583 0.14038 0 - 0.36018 2024-04-22T06:42:19.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1596 16 3599 3583 0.140292 0 - 0.36018 2024-04-22T06:42:20.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1597 16 3599 3583 0.140204 0 - 0.36018 2024-04-22T06:42:21.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1598 16 3599 3583 0.140116 0 - 0.36018 2024-04-22T06:42:22.676 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1599 16 3599 3583 0.140028 0 - 0.36018 2024-04-22T06:42:23.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:42:23.679568+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:42:23.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:42:23.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1600 16 3599 3583 0.139941 0 - 0.36018 2024-04-22T06:42:24.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1601 16 3599 3583 0.139853 0 - 0.36018 2024-04-22T06:42:25.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1602 16 3599 3583 0.139766 0 - 0.36018 2024-04-22T06:42:26.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1603 16 3599 3583 0.139679 0 - 0.36018 2024-04-22T06:42:27.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1604 16 3599 3583 0.139592 0 - 0.36018 2024-04-22T06:42:28.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1605 16 3599 3583 0.139505 0 - 0.36018 2024-04-22T06:42:29.677 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1606 16 3599 3583 0.139418 0 - 0.36018 2024-04-22T06:42:30.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1607 16 3599 3583 0.139331 0 - 0.36018 2024-04-22T06:42:31.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1608 16 3599 3583 0.139245 0 - 0.36018 2024-04-22T06:42:32.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1609 16 3599 3583 0.139158 0 - 0.36018 2024-04-22T06:42:33.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1610 16 3599 3583 0.139072 0 - 0.36018 2024-04-22T06:42:34.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1611 16 3599 3583 0.138985 0 - 0.36018 2024-04-22T06:42:35.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1612 16 3599 3583 0.138899 0 - 0.36018 2024-04-22T06:42:36.678 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1613 16 3599 3583 0.138813 0 - 0.36018 2024-04-22T06:42:37.679 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1614 16 3599 3583 0.138727 0 - 0.36018 2024-04-22T06:42:38.679 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1615 16 3599 3583 0.138641 0 - 0.36018 2024-04-22T06:42:39.679 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1616 16 3599 3583 0.138555 0 - 0.36018 2024-04-22T06:42:40.679 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1617 16 3599 3583 0.13847 0 - 0.36018 2024-04-22T06:42:41.679 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1618 16 3599 3583 0.138384 0 - 0.36018 2024-04-22T06:42:42.679 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1619 16 3599 3583 0.138299 0 - 0.36018 2024-04-22T06:42:43.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:42:43.682642+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:42:43.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:42:43.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1620 16 3599 3583 0.138213 0 - 0.36018 2024-04-22T06:42:44.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1621 16 3599 3583 0.138128 0 - 0.36018 2024-04-22T06:42:45.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1622 16 3599 3583 0.138043 0 - 0.36018 2024-04-22T06:42:46.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1623 16 3599 3583 0.137958 0 - 0.36018 2024-04-22T06:42:47.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1624 16 3599 3583 0.137873 0 - 0.36018 2024-04-22T06:42:48.680 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1625 16 3599 3583 0.137788 0 - 0.36018 2024-04-22T06:42:49.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1626 16 3599 3583 0.137703 0 - 0.36018 2024-04-22T06:42:50.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1627 16 3599 3583 0.137619 0 - 0.36018 2024-04-22T06:42:51.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1628 16 3599 3583 0.137534 0 - 0.36018 2024-04-22T06:42:52.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1629 16 3599 3583 0.13745 0 - 0.36018 2024-04-22T06:42:53.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1630 16 3599 3583 0.137365 0 - 0.36018 2024-04-22T06:42:54.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1631 16 3599 3583 0.137281 0 - 0.36018 2024-04-22T06:42:55.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1632 16 3599 3583 0.137197 0 - 0.36018 2024-04-22T06:42:56.681 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1633 16 3599 3583 0.137113 0 - 0.36018 2024-04-22T06:42:57.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1634 16 3599 3583 0.137029 0 - 0.36018 2024-04-22T06:42:58.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1635 16 3599 3583 0.136945 0 - 0.36018 2024-04-22T06:42:59.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1636 16 3599 3583 0.136861 0 - 0.36018 2024-04-22T06:43:00.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1637 16 3599 3583 0.136778 0 - 0.36018 2024-04-22T06:43:01.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1638 16 3599 3583 0.136694 0 - 0.36018 2024-04-22T06:43:02.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1639 16 3599 3583 0.136611 0 - 0.36018 2024-04-22T06:43:03.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:43:03.685431+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:43:03.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:43:03.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1640 16 3599 3583 0.136528 0 - 0.36018 2024-04-22T06:43:04.682 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1641 16 3599 3583 0.136444 0 - 0.36018 2024-04-22T06:43:05.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1642 16 3599 3583 0.136361 0 - 0.36018 2024-04-22T06:43:06.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1643 16 3599 3583 0.136278 0 - 0.36018 2024-04-22T06:43:07.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1644 16 3599 3583 0.136195 0 - 0.36018 2024-04-22T06:43:08.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1645 16 3599 3583 0.136113 0 - 0.36018 2024-04-22T06:43:09.683 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1646 16 3599 3583 0.13603 0 - 0.36018 2024-04-22T06:43:10.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1647 16 3599 3583 0.135947 0 - 0.36018 2024-04-22T06:43:11.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1648 16 3599 3583 0.135865 0 - 0.36018 2024-04-22T06:43:12.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1649 16 3599 3583 0.135782 0 - 0.36018 2024-04-22T06:43:13.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1650 16 3599 3583 0.1357 0 - 0.36018 2024-04-22T06:43:14.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1651 16 3599 3583 0.135618 0 - 0.36018 2024-04-22T06:43:15.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1652 16 3599 3583 0.135536 0 - 0.36018 2024-04-22T06:43:16.684 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1653 16 3599 3583 0.135454 0 - 0.36018 2024-04-22T06:43:17.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1654 16 3599 3583 0.135372 0 - 0.36018 2024-04-22T06:43:18.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1655 16 3599 3583 0.13529 0 - 0.36018 2024-04-22T06:43:19.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1656 16 3599 3583 0.135209 0 - 0.36018 2024-04-22T06:43:20.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1657 16 3599 3583 0.135127 0 - 0.36018 2024-04-22T06:43:21.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1658 16 3599 3583 0.135045 0 - 0.36018 2024-04-22T06:43:22.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1659 16 3599 3583 0.134964 0 - 0.36018 2024-04-22T06:43:23.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:43:23.688340+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:43:23.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:43:23.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1660 16 3599 3583 0.134883 0 - 0.36018 2024-04-22T06:43:24.685 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1661 16 3599 3583 0.134802 0 - 0.36018 2024-04-22T06:43:25.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1662 16 3599 3583 0.13472 0 - 0.36018 2024-04-22T06:43:26.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1663 16 3599 3583 0.134639 0 - 0.36018 2024-04-22T06:43:27.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1664 16 3599 3583 0.134558 0 - 0.36018 2024-04-22T06:43:28.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1665 16 3599 3583 0.134478 0 - 0.36018 2024-04-22T06:43:29.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1666 16 3599 3583 0.134397 0 - 0.36018 2024-04-22T06:43:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:43:30.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:43:30.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1667 16 3599 3583 0.134316 0 - 0.36018 2024-04-22T06:43:31.686 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1668 16 3599 3583 0.134236 0 - 0.36018 2024-04-22T06:43:32.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:43:32.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:43:32.396+0000 7f3b8c006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:43:32.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1669 16 3599 3583 0.134155 0 - 0.36018 2024-04-22T06:43:33.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1670 16 3599 3583 0.134075 0 - 0.36018 2024-04-22T06:43:34.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1671 16 3599 3583 0.133995 0 - 0.36018 2024-04-22T06:43:35.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1672 16 3599 3583 0.133915 0 - 0.36018 2024-04-22T06:43:36.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1673 16 3599 3583 0.133835 0 - 0.36018 2024-04-22T06:43:37.687 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1674 16 3599 3583 0.133755 0 - 0.36018 2024-04-22T06:43:38.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1675 16 3599 3583 0.133675 0 - 0.36018 2024-04-22T06:43:39.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1676 16 3599 3583 0.133595 0 - 0.36018 2024-04-22T06:43:40.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1677 16 3599 3583 0.133515 0 - 0.36018 2024-04-22T06:43:41.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1678 16 3599 3583 0.133436 0 - 0.36018 2024-04-22T06:43:42.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1679 16 3599 3583 0.133356 0 - 0.36018 2024-04-22T06:43:43.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:43:43.691361+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:43:43.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:43:43.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1680 16 3599 3583 0.133277 0 - 0.36018 2024-04-22T06:43:44.688 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1681 16 3599 3583 0.133198 0 - 0.36018 2024-04-22T06:43:45.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1682 16 3599 3583 0.133118 0 - 0.36018 2024-04-22T06:43:46.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1683 16 3599 3583 0.133039 0 - 0.36018 2024-04-22T06:43:47.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1684 16 3599 3583 0.13296 0 - 0.36018 2024-04-22T06:43:48.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1685 16 3599 3583 0.132881 0 - 0.36018 2024-04-22T06:43:49.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1686 16 3599 3583 0.132803 0 - 0.36018 2024-04-22T06:43:50.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1687 16 3599 3583 0.132724 0 - 0.36018 2024-04-22T06:43:51.689 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1688 16 3599 3583 0.132645 0 - 0.36018 2024-04-22T06:43:52.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1689 16 3599 3583 0.132567 0 - 0.36018 2024-04-22T06:43:53.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1690 16 3599 3583 0.132488 0 - 0.36018 2024-04-22T06:43:54.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1691 16 3599 3583 0.13241 0 - 0.36018 2024-04-22T06:43:55.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1692 16 3599 3583 0.132332 0 - 0.36018 2024-04-22T06:43:56.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1693 16 3599 3583 0.132254 0 - 0.36018 2024-04-22T06:43:57.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1694 16 3599 3583 0.132175 0 - 0.36018 2024-04-22T06:43:58.690 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1695 16 3599 3583 0.132098 0 - 0.36018 2024-04-22T06:43:59.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1696 16 3599 3583 0.13202 0 - 0.36018 2024-04-22T06:44:00.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1697 16 3599 3583 0.131942 0 - 0.36018 2024-04-22T06:44:01.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1698 16 3599 3583 0.131864 0 - 0.36018 2024-04-22T06:44:02.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1699 16 3599 3583 0.131787 0 - 0.36018 2024-04-22T06:44:03.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:44:03.694317+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:44:03.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:44:03.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1700 16 3599 3583 0.131709 0 - 0.36018 2024-04-22T06:44:04.691 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1701 16 3599 3583 0.131632 0 - 0.36018 2024-04-22T06:44:05.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1702 16 3599 3583 0.131554 0 - 0.36018 2024-04-22T06:44:06.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1703 16 3599 3583 0.131477 0 - 0.36018 2024-04-22T06:44:07.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1704 16 3599 3583 0.1314 0 - 0.36018 2024-04-22T06:44:08.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1705 16 3599 3583 0.131323 0 - 0.36018 2024-04-22T06:44:09.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1706 16 3599 3583 0.131246 0 - 0.36018 2024-04-22T06:44:10.692 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1707 16 3599 3583 0.131169 0 - 0.36018 2024-04-22T06:44:11.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1708 16 3599 3583 0.131092 0 - 0.36018 2024-04-22T06:44:12.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1709 16 3599 3583 0.131015 0 - 0.36018 2024-04-22T06:44:13.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1710 16 3599 3583 0.130939 0 - 0.36018 2024-04-22T06:44:14.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1711 16 3599 3583 0.130862 0 - 0.36018 2024-04-22T06:44:15.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1712 16 3599 3583 0.130786 0 - 0.36018 2024-04-22T06:44:16.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1713 16 3599 3583 0.130709 0 - 0.36018 2024-04-22T06:44:17.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1714 16 3599 3583 0.130633 0 - 0.36018 2024-04-22T06:44:18.693 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1715 16 3599 3583 0.130557 0 - 0.36018 2024-04-22T06:44:19.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1716 16 3599 3583 0.130481 0 - 0.36018 2024-04-22T06:44:20.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1717 16 3599 3583 0.130405 0 - 0.36018 2024-04-22T06:44:21.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1718 16 3599 3583 0.130329 0 - 0.36018 2024-04-22T06:44:22.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1719 16 3599 3583 0.130253 0 - 0.36018 2024-04-22T06:44:23.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:44:23.697253+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:44:23.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:44:23.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1720 16 3599 3583 0.130177 0 - 0.36018 2024-04-22T06:44:24.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1721 16 3599 3583 0.130102 0 - 0.36018 2024-04-22T06:44:25.694 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1722 16 3599 3583 0.130026 0 - 0.36018 2024-04-22T06:44:26.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1723 16 3599 3583 0.129951 0 - 0.36018 2024-04-22T06:44:27.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1724 16 3599 3583 0.129875 0 - 0.36018 2024-04-22T06:44:28.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1725 16 3599 3583 0.1298 0 - 0.36018 2024-04-22T06:44:29.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1726 16 3599 3583 0.129725 0 - 0.36018 2024-04-22T06:44:30.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1727 16 3599 3583 0.12965 0 - 0.36018 2024-04-22T06:44:31.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1728 16 3599 3583 0.129575 0 - 0.36018 2024-04-22T06:44:32.695 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1729 16 3599 3583 0.1295 0 - 0.36018 2024-04-22T06:44:33.696 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1730 16 3599 3583 0.129425 0 - 0.36018 2024-04-22T06:44:34.696 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1731 16 3599 3583 0.12935 0 - 0.36018 2024-04-22T06:44:35.696 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1732 16 3599 3583 0.129276 0 - 0.36018 2024-04-22T06:44:36.696 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1733 16 3599 3583 0.129201 0 - 0.36018 2024-04-22T06:44:37.696 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1734 16 3599 3583 0.129126 0 - 0.36018 2024-04-22T06:44:38.696 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1735 16 3599 3583 0.129052 0 - 0.36018 2024-04-22T06:44:39.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1736 16 3599 3583 0.128978 0 - 0.36018 2024-04-22T06:44:40.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1737 16 3599 3583 0.128903 0 - 0.36018 2024-04-22T06:44:41.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1738 16 3599 3583 0.128829 0 - 0.36018 2024-04-22T06:44:42.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1739 16 3599 3583 0.128755 0 - 0.36018 2024-04-22T06:44:43.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:44:43.700130+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:44:43.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:44:43.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1740 16 3599 3583 0.128681 0 - 0.36018 2024-04-22T06:44:44.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1741 16 3599 3583 0.128607 0 - 0.36018 2024-04-22T06:44:45.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1742 16 3599 3583 0.128533 0 - 0.36018 2024-04-22T06:44:46.697 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1743 16 3599 3583 0.12846 0 - 0.36018 2024-04-22T06:44:47.698 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1744 16 3599 3583 0.128386 0 - 0.36018 2024-04-22T06:44:48.698 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1745 16 3599 3583 0.128312 0 - 0.36018 2024-04-22T06:44:49.698 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1746 16 3599 3583 0.128239 0 - 0.36018 2024-04-22T06:44:50.698 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1747 16 3599 3583 0.128166 0 - 0.36018 2024-04-22T06:44:51.698 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1748 16 3599 3583 0.128092 0 - 0.36018 2024-04-22T06:44:52.698 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1749 16 3599 3583 0.128019 0 - 0.36018 2024-04-22T06:44:53.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1750 16 3599 3583 0.127946 0 - 0.36018 2024-04-22T06:44:54.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1751 16 3599 3583 0.127873 0 - 0.36018 2024-04-22T06:44:55.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1752 16 3599 3583 0.1278 0 - 0.36018 2024-04-22T06:44:56.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1753 16 3599 3583 0.127727 0 - 0.36018 2024-04-22T06:44:57.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1754 16 3599 3583 0.127654 0 - 0.36018 2024-04-22T06:44:58.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1755 16 3599 3583 0.127581 0 - 0.36018 2024-04-22T06:44:59.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1756 16 3599 3583 0.127509 0 - 0.36018 2024-04-22T06:45:00.699 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1757 16 3599 3583 0.127436 0 - 0.36018 2024-04-22T06:45:01.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1758 16 3599 3583 0.127364 0 - 0.36018 2024-04-22T06:45:02.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1759 16 3599 3583 0.127291 0 - 0.36018 2024-04-22T06:45:03.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:45:03.702942+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:45:03.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:45:03.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1760 16 3599 3583 0.127219 0 - 0.36018 2024-04-22T06:45:04.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1761 16 3599 3583 0.127147 0 - 0.36018 2024-04-22T06:45:05.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1762 16 3599 3583 0.127075 0 - 0.36018 2024-04-22T06:45:06.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1763 16 3599 3583 0.127002 0 - 0.36018 2024-04-22T06:45:07.700 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1764 16 3599 3583 0.12693 0 - 0.36018 2024-04-22T06:45:08.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1765 16 3599 3583 0.126859 0 - 0.36018 2024-04-22T06:45:09.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1766 16 3599 3583 0.126787 0 - 0.36018 2024-04-22T06:45:10.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1767 16 3599 3583 0.126715 0 - 0.36018 2024-04-22T06:45:11.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1768 16 3599 3583 0.126643 0 - 0.36018 2024-04-22T06:45:12.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1769 16 3599 3583 0.126572 0 - 0.36018 2024-04-22T06:45:13.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1770 16 3599 3583 0.1265 0 - 0.36018 2024-04-22T06:45:14.701 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1771 16 3599 3583 0.126429 0 - 0.36018 2024-04-22T06:45:15.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1772 16 3599 3583 0.126357 0 - 0.36018 2024-04-22T06:45:16.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1773 16 3599 3583 0.126286 0 - 0.36018 2024-04-22T06:45:17.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1774 16 3599 3583 0.126215 0 - 0.36018 2024-04-22T06:45:18.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1775 16 3599 3583 0.126144 0 - 0.36018 2024-04-22T06:45:19.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1776 16 3599 3583 0.126073 0 - 0.36018 2024-04-22T06:45:20.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1777 16 3599 3583 0.126002 0 - 0.36018 2024-04-22T06:45:21.702 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1778 16 3599 3583 0.125931 0 - 0.36018 2024-04-22T06:45:22.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1779 16 3599 3583 0.12586 0 - 0.36018 2024-04-22T06:45:23.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:45:23.705743+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:45:23.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:45:23.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1780 16 3599 3583 0.125789 0 - 0.36018 2024-04-22T06:45:24.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1781 16 3599 3583 0.125719 0 - 0.36018 2024-04-22T06:45:25.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1782 16 3599 3583 0.125648 0 - 0.36018 2024-04-22T06:45:26.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1783 16 3599 3583 0.125578 0 - 0.36018 2024-04-22T06:45:27.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1784 16 3599 3583 0.125507 0 - 0.36018 2024-04-22T06:45:28.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1785 16 3599 3583 0.125437 0 - 0.36018 2024-04-22T06:45:29.703 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1786 16 3599 3583 0.125367 0 - 0.36018 2024-04-22T06:45:30.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1787 16 3599 3583 0.125297 0 - 0.36018 2024-04-22T06:45:31.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1788 16 3599 3583 0.125227 0 - 0.36018 2024-04-22T06:45:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:45:32.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:45:32.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1789 16 3599 3583 0.125157 0 - 0.36018 2024-04-22T06:45:33.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1790 16 3599 3583 0.125087 0 - 0.36018 2024-04-22T06:45:34.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:45:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:45:34.433+0000 7f9883257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:45:34.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1791 16 3599 3583 0.125017 0 - 0.36018 2024-04-22T06:45:35.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1792 16 3599 3583 0.124947 0 - 0.36018 2024-04-22T06:45:36.704 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1793 16 3599 3583 0.124877 0 - 0.36018 2024-04-22T06:45:37.705 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1794 16 3599 3583 0.124808 0 - 0.36018 2024-04-22T06:45:38.705 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1795 16 3599 3583 0.124738 0 - 0.36018 2024-04-22T06:45:39.705 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1796 16 3599 3583 0.124669 0 - 0.36018 2024-04-22T06:45:40.705 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1797 16 3599 3583 0.124599 0 - 0.36018 2024-04-22T06:45:41.705 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1798 16 3599 3583 0.12453 0 - 0.36018 2024-04-22T06:45:42.705 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1799 16 3599 3583 0.124461 0 - 0.36018 2024-04-22T06:45:43.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:45:43.708639+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:45:43.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:45:43.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1800 16 3599 3583 0.124392 0 - 0.36018 2024-04-22T06:45:44.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1801 16 3599 3583 0.124323 0 - 0.36018 2024-04-22T06:45:45.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1802 16 3599 3583 0.124254 0 - 0.36018 2024-04-22T06:45:46.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1803 16 3599 3583 0.124185 0 - 0.36018 2024-04-22T06:45:47.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1804 16 3599 3583 0.124116 0 - 0.36018 2024-04-22T06:45:48.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1805 16 3599 3583 0.124047 0 - 0.36018 2024-04-22T06:45:49.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1806 16 3599 3583 0.123979 0 - 0.36018 2024-04-22T06:45:50.706 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1807 16 3599 3583 0.12391 0 - 0.36018 2024-04-22T06:45:51.707 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1808 16 3599 3583 0.123841 0 - 0.36018 2024-04-22T06:45:52.707 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1809 16 3599 3583 0.123773 0 - 0.36018 2024-04-22T06:45:53.707 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1810 16 3599 3583 0.123705 0 - 0.36018 2024-04-22T06:45:54.707 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1811 16 3599 3583 0.123636 0 - 0.36018 2024-04-22T06:45:55.707 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1812 16 3599 3583 0.123568 0 - 0.36018 2024-04-22T06:45:56.707 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1813 16 3599 3583 0.1235 0 - 0.36018 2024-04-22T06:45:57.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1814 16 3599 3583 0.123432 0 - 0.36018 2024-04-22T06:45:58.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1815 16 3599 3583 0.123364 0 - 0.36018 2024-04-22T06:45:59.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1816 16 3599 3583 0.123296 0 - 0.36018 2024-04-22T06:46:00.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1817 16 3599 3583 0.123228 0 - 0.36018 2024-04-22T06:46:01.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1818 16 3599 3583 0.12316 0 - 0.36018 2024-04-22T06:46:02.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1819 16 3599 3583 0.123093 0 - 0.36018 2024-04-22T06:46:03.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:46:03.711436+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:46:03.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:46:03.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1820 16 3599 3583 0.123025 0 - 0.36018 2024-04-22T06:46:04.708 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1821 16 3599 3583 0.122957 0 - 0.36018 2024-04-22T06:46:05.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1822 16 3599 3583 0.12289 0 - 0.36018 2024-04-22T06:46:06.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1823 16 3599 3583 0.122822 0 - 0.36018 2024-04-22T06:46:07.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1824 16 3599 3583 0.122755 0 - 0.36018 2024-04-22T06:46:08.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1825 16 3599 3583 0.122688 0 - 0.36018 2024-04-22T06:46:09.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1826 16 3599 3583 0.122621 0 - 0.36018 2024-04-22T06:46:10.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1827 16 3599 3583 0.122554 0 - 0.36018 2024-04-22T06:46:11.709 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1828 16 3599 3583 0.122486 0 - 0.36018 2024-04-22T06:46:12.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1829 16 3599 3583 0.12242 0 - 0.36018 2024-04-22T06:46:13.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1830 16 3599 3583 0.122353 0 - 0.36018 2024-04-22T06:46:14.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1831 16 3599 3583 0.122286 0 - 0.36018 2024-04-22T06:46:15.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1832 16 3599 3583 0.122219 0 - 0.36018 2024-04-22T06:46:16.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1833 16 3599 3583 0.122152 0 - 0.36018 2024-04-22T06:46:17.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1834 16 3599 3583 0.122086 0 - 0.36018 2024-04-22T06:46:18.710 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1835 16 3599 3583 0.122019 0 - 0.36018 2024-04-22T06:46:19.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1836 16 3599 3583 0.121953 0 - 0.36018 2024-04-22T06:46:20.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1837 16 3599 3583 0.121886 0 - 0.36018 2024-04-22T06:46:21.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1838 16 3599 3583 0.12182 0 - 0.36018 2024-04-22T06:46:22.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1839 16 3599 3583 0.121754 0 - 0.36018 2024-04-22T06:46:23.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:46:23.714446+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:46:23.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:46:23.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1840 16 3599 3583 0.121688 0 - 0.36018 2024-04-22T06:46:24.711 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1841 16 3599 3583 0.121622 0 - 0.36018 2024-04-22T06:46:25.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1842 16 3599 3583 0.121556 0 - 0.36018 2024-04-22T06:46:26.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1843 16 3599 3583 0.12149 0 - 0.36018 2024-04-22T06:46:27.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1844 16 3599 3583 0.121424 0 - 0.36018 2024-04-22T06:46:28.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1845 16 3599 3583 0.121358 0 - 0.36018 2024-04-22T06:46:29.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1846 16 3599 3583 0.121292 0 - 0.36018 2024-04-22T06:46:30.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1847 16 3599 3583 0.121226 0 - 0.36018 2024-04-22T06:46:31.712 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1848 16 3599 3583 0.121161 0 - 0.36018 2024-04-22T06:46:32.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1849 16 3599 3583 0.121095 0 - 0.36018 2024-04-22T06:46:33.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1850 16 3599 3583 0.12103 0 - 0.36018 2024-04-22T06:46:34.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1851 16 3599 3583 0.120964 0 - 0.36018 2024-04-22T06:46:35.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1852 16 3599 3583 0.120899 0 - 0.36018 2024-04-22T06:46:36.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1853 16 3599 3583 0.120834 0 - 0.36018 2024-04-22T06:46:37.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1854 16 3599 3583 0.120769 0 - 0.36018 2024-04-22T06:46:38.713 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1855 16 3599 3583 0.120704 0 - 0.36018 2024-04-22T06:46:39.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1856 16 3599 3583 0.120639 0 - 0.36018 2024-04-22T06:46:40.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1857 16 3599 3583 0.120574 0 - 0.36018 2024-04-22T06:46:41.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1858 16 3599 3583 0.120509 0 - 0.36018 2024-04-22T06:46:42.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1859 16 3599 3583 0.120444 0 - 0.36018 2024-04-22T06:46:43.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:46:43.717328+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:46:43.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:46:43.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1860 16 3599 3583 0.120379 0 - 0.36018 2024-04-22T06:46:44.714 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1861 16 3599 3583 0.120314 0 - 0.36018 2024-04-22T06:46:45.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1862 16 3599 3583 0.12025 0 - 0.36018 2024-04-22T06:46:46.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1863 16 3599 3583 0.120185 0 - 0.36018 2024-04-22T06:46:47.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1864 16 3599 3583 0.120121 0 - 0.36018 2024-04-22T06:46:48.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1865 16 3599 3583 0.120056 0 - 0.36018 2024-04-22T06:46:49.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1866 16 3599 3583 0.119992 0 - 0.36018 2024-04-22T06:46:50.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1867 16 3599 3583 0.119928 0 - 0.36018 2024-04-22T06:46:51.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1868 16 3599 3583 0.119864 0 - 0.36018 2024-04-22T06:46:52.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1869 16 3599 3583 0.1198 0 - 0.36018 2024-04-22T06:46:53.715 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1870 16 3599 3583 0.119735 0 - 0.36018 2024-04-22T06:46:54.716 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1871 16 3599 3583 0.119671 0 - 0.36018 2024-04-22T06:46:55.716 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1872 16 3599 3583 0.119608 0 - 0.36018 2024-04-22T06:46:56.716 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1873 16 3599 3583 0.119544 0 - 0.36018 2024-04-22T06:46:57.716 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1874 16 3599 3583 0.11948 0 - 0.36018 2024-04-22T06:46:58.716 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1875 16 3599 3583 0.119416 0 - 0.36018 2024-04-22T06:46:59.716 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1876 16 3599 3583 0.119352 0 - 0.36018 2024-04-22T06:47:00.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1877 16 3599 3583 0.119289 0 - 0.36018 2024-04-22T06:47:01.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1878 16 3599 3583 0.119225 0 - 0.36018 2024-04-22T06:47:02.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1879 16 3599 3583 0.119162 0 - 0.36018 2024-04-22T06:47:03.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:47:03.720223+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:47:03.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:47:03.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1880 16 3599 3583 0.119099 0 - 0.36018 2024-04-22T06:47:04.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1881 16 3599 3583 0.119035 0 - 0.36018 2024-04-22T06:47:05.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1882 16 3599 3583 0.118972 0 - 0.36018 2024-04-22T06:47:06.717 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1883 16 3599 3583 0.118909 0 - 0.36018 2024-04-22T06:47:07.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1884 16 3599 3583 0.118846 0 - 0.36018 2024-04-22T06:47:08.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1885 16 3599 3583 0.118783 0 - 0.36018 2024-04-22T06:47:09.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1886 16 3599 3583 0.11872 0 - 0.36018 2024-04-22T06:47:10.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1887 16 3599 3583 0.118657 0 - 0.36018 2024-04-22T06:47:11.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1888 16 3599 3583 0.118594 0 - 0.36018 2024-04-22T06:47:12.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1889 16 3599 3583 0.118531 0 - 0.36018 2024-04-22T06:47:13.718 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1890 16 3599 3583 0.118468 0 - 0.36018 2024-04-22T06:47:14.719 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1891 16 3599 3583 0.118406 0 - 0.36018 2024-04-22T06:47:15.719 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1892 16 3599 3583 0.118343 0 - 0.36018 2024-04-22T06:47:16.719 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1893 16 3599 3583 0.118281 0 - 0.36018 2024-04-22T06:47:17.719 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1894 16 3599 3583 0.118218 0 - 0.36018 2024-04-22T06:47:18.719 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1895 16 3599 3583 0.118156 0 - 0.36018 2024-04-22T06:47:19.719 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1896 16 3599 3583 0.118094 0 - 0.36018 2024-04-22T06:47:20.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1897 16 3599 3583 0.118031 0 - 0.36018 2024-04-22T06:47:21.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1898 16 3599 3583 0.117969 0 - 0.36018 2024-04-22T06:47:22.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1899 16 3599 3583 0.117907 0 - 0.36018 2024-04-22T06:47:23.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:47:23.723098+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:47:23.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:47:23.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1900 16 3599 3583 0.117845 0 - 0.36018 2024-04-22T06:47:24.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1901 16 3599 3583 0.117783 0 - 0.36018 2024-04-22T06:47:25.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1902 16 3599 3583 0.117721 0 - 0.36018 2024-04-22T06:47:26.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1903 16 3599 3583 0.117659 0 - 0.36018 2024-04-22T06:47:27.720 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1904 16 3599 3583 0.117597 0 - 0.36018 2024-04-22T06:47:28.721 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1905 16 3599 3583 0.117536 0 - 0.36018 2024-04-22T06:47:29.721 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1906 16 3599 3583 0.117474 0 - 0.36018 2024-04-22T06:47:30.721 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1907 16 3599 3583 0.117412 0 - 0.36018 2024-04-22T06:47:31.721 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1908 16 3599 3583 0.117351 0 - 0.36018 2024-04-22T06:47:32.721 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1909 16 3599 3583 0.117289 0 - 0.36018 2024-04-22T06:47:33.721 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1910 16 3599 3583 0.117228 0 - 0.36018 2024-04-22T06:47:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:47:34.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:47:34.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1911 16 3599 3583 0.117167 0 - 0.36018 2024-04-22T06:47:35.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1912 16 3599 3583 0.117105 0 - 0.36018 2024-04-22T06:47:36.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:47:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:47:36.470+0000 7f5769424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:47:36.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1913 16 3599 3583 0.117044 0 - 0.36018 2024-04-22T06:47:37.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1914 16 3599 3583 0.116983 0 - 0.36018 2024-04-22T06:47:38.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1915 16 3599 3583 0.116922 0 - 0.36018 2024-04-22T06:47:39.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1916 16 3599 3583 0.116861 0 - 0.36018 2024-04-22T06:47:40.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1917 16 3599 3583 0.1168 0 - 0.36018 2024-04-22T06:47:41.722 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1918 16 3599 3583 0.116739 0 - 0.36018 2024-04-22T06:47:42.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1919 16 3599 3583 0.116678 0 - 0.36018 2024-04-22T06:47:43.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:47:43.725919+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:47:43.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:47:43.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1920 16 3599 3583 0.116617 0 - 0.36018 2024-04-22T06:47:44.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1921 16 3599 3583 0.116557 0 - 0.36018 2024-04-22T06:47:45.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1922 16 3599 3583 0.116496 0 - 0.36018 2024-04-22T06:47:46.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1923 16 3599 3583 0.116435 0 - 0.36018 2024-04-22T06:47:47.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1924 16 3599 3583 0.116375 0 - 0.36018 2024-04-22T06:47:48.723 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1925 16 3599 3583 0.116314 0 - 0.36018 2024-04-22T06:47:49.724 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1926 16 3599 3583 0.116254 0 - 0.36018 2024-04-22T06:47:50.724 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1927 16 3599 3583 0.116194 0 - 0.36018 2024-04-22T06:47:51.724 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1928 16 3599 3583 0.116133 0 - 0.36018 2024-04-22T06:47:52.724 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1929 16 3599 3583 0.116073 0 - 0.36018 2024-04-22T06:47:53.724 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1930 16 3599 3583 0.116013 0 - 0.36018 2024-04-22T06:47:54.724 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1931 16 3599 3583 0.115953 0 - 0.36018 2024-04-22T06:47:55.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1932 16 3599 3583 0.115893 0 - 0.36018 2024-04-22T06:47:56.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1933 16 3599 3583 0.115833 0 - 0.36018 2024-04-22T06:47:57.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1934 16 3599 3583 0.115773 0 - 0.36018 2024-04-22T06:47:58.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1935 16 3599 3583 0.115713 0 - 0.36018 2024-04-22T06:47:59.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1936 16 3599 3583 0.115654 0 - 0.36018 2024-04-22T06:48:00.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1937 16 3599 3583 0.115594 0 - 0.36018 2024-04-22T06:48:01.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1938 16 3599 3583 0.115534 0 - 0.36018 2024-04-22T06:48:02.725 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1939 16 3599 3583 0.115475 0 - 0.36018 2024-04-22T06:48:03.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:48:03.728818+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:48:03.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:48:03.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1940 16 3599 3583 0.115415 0 - 0.36018 2024-04-22T06:48:04.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1941 16 3599 3583 0.115356 0 - 0.36018 2024-04-22T06:48:05.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1942 16 3599 3583 0.115296 0 - 0.36018 2024-04-22T06:48:06.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1943 16 3599 3583 0.115237 0 - 0.36018 2024-04-22T06:48:07.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1944 16 3599 3583 0.115178 0 - 0.36018 2024-04-22T06:48:08.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1945 16 3599 3583 0.115118 0 - 0.36018 2024-04-22T06:48:09.726 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1946 16 3599 3583 0.115059 0 - 0.36018 2024-04-22T06:48:10.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1947 16 3599 3583 0.115 0 - 0.36018 2024-04-22T06:48:11.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1948 16 3599 3583 0.114941 0 - 0.36018 2024-04-22T06:48:12.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1949 16 3599 3583 0.114882 0 - 0.36018 2024-04-22T06:48:13.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1950 16 3599 3583 0.114823 0 - 0.36018 2024-04-22T06:48:14.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1951 16 3599 3583 0.114764 0 - 0.36018 2024-04-22T06:48:15.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1952 16 3599 3583 0.114706 0 - 0.36018 2024-04-22T06:48:16.727 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1953 16 3599 3583 0.114647 0 - 0.36018 2024-04-22T06:48:17.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1954 16 3599 3583 0.114588 0 - 0.36018 2024-04-22T06:48:18.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1955 16 3599 3583 0.11453 0 - 0.36018 2024-04-22T06:48:19.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1956 16 3599 3583 0.114471 0 - 0.36018 2024-04-22T06:48:20.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1957 16 3599 3583 0.114413 0 - 0.36018 2024-04-22T06:48:21.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1958 16 3599 3583 0.114354 0 - 0.36018 2024-04-22T06:48:22.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1959 16 3599 3583 0.114296 0 - 0.36018 2024-04-22T06:48:23.728 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:48:23.731620+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:48:23.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:48:23.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1960 16 3599 3583 0.114237 0 - 0.36018 2024-04-22T06:48:24.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1961 16 3599 3583 0.114179 0 - 0.36018 2024-04-22T06:48:25.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1962 16 3599 3583 0.114121 0 - 0.36018 2024-04-22T06:48:26.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1963 16 3599 3583 0.114063 0 - 0.36018 2024-04-22T06:48:27.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1964 16 3599 3583 0.114005 0 - 0.36018 2024-04-22T06:48:28.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1965 16 3599 3583 0.113947 0 - 0.36018 2024-04-22T06:48:29.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1966 16 3599 3583 0.113889 0 - 0.36018 2024-04-22T06:48:30.729 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1967 16 3599 3583 0.113831 0 - 0.36018 2024-04-22T06:48:31.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1968 16 3599 3583 0.113773 0 - 0.36018 2024-04-22T06:48:32.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1969 16 3599 3583 0.113715 0 - 0.36018 2024-04-22T06:48:33.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1970 16 3599 3583 0.113658 0 - 0.36018 2024-04-22T06:48:34.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1971 16 3599 3583 0.1136 0 - 0.36018 2024-04-22T06:48:35.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1972 16 3599 3583 0.113542 0 - 0.36018 2024-04-22T06:48:36.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1973 16 3599 3583 0.113485 0 - 0.36018 2024-04-22T06:48:37.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1974 16 3599 3583 0.113427 0 - 0.36018 2024-04-22T06:48:38.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1975 16 3599 3583 0.11337 0 - 0.36018 2024-04-22T06:48:39.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1976 16 3599 3583 0.113312 0 - 0.36018 2024-04-22T06:48:40.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1977 16 3599 3583 0.113255 0 - 0.36018 2024-04-22T06:48:41.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1978 16 3599 3583 0.113198 0 - 0.36018 2024-04-22T06:48:42.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1979 16 3599 3583 0.113141 0 - 0.36018 2024-04-22T06:48:43.731 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:48:43.734615+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:48:43.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:48:43.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1980 16 3599 3583 0.113083 0 - 0.36018 2024-04-22T06:48:44.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1981 16 3599 3583 0.113026 0 - 0.36018 2024-04-22T06:48:45.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1982 16 3599 3583 0.112969 0 - 0.36018 2024-04-22T06:48:46.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1983 16 3599 3583 0.112912 0 - 0.36018 2024-04-22T06:48:47.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1984 16 3599 3583 0.112855 0 - 0.36018 2024-04-22T06:48:48.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1985 16 3599 3583 0.112799 0 - 0.36018 2024-04-22T06:48:49.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1986 16 3599 3583 0.112742 0 - 0.36018 2024-04-22T06:48:50.732 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1987 16 3599 3583 0.112685 0 - 0.36018 2024-04-22T06:48:51.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1988 16 3599 3583 0.112628 0 - 0.36018 2024-04-22T06:48:52.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1989 16 3599 3583 0.112572 0 - 0.36018 2024-04-22T06:48:53.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1990 16 3599 3583 0.112515 0 - 0.36018 2024-04-22T06:48:54.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1991 16 3599 3583 0.112459 0 - 0.36018 2024-04-22T06:48:55.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1992 16 3599 3583 0.112402 0 - 0.36018 2024-04-22T06:48:56.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1993 16 3599 3583 0.112346 0 - 0.36018 2024-04-22T06:48:57.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1994 16 3599 3583 0.112289 0 - 0.36018 2024-04-22T06:48:58.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1995 16 3599 3583 0.112233 0 - 0.36018 2024-04-22T06:48:59.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1996 16 3599 3583 0.112177 0 - 0.36018 2024-04-22T06:49:00.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1997 16 3599 3583 0.112121 0 - 0.36018 2024-04-22T06:49:01.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1998 16 3599 3583 0.112065 0 - 0.36018 2024-04-22T06:49:02.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1999 16 3599 3583 0.112009 0 - 0.36018 2024-04-22T06:49:03.734 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:49:03.737647+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:49:03.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:49:03.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2000 16 3599 3583 0.111953 0 - 0.36018 2024-04-22T06:49:04.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2001 16 3599 3583 0.111897 0 - 0.36018 2024-04-22T06:49:05.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2002 16 3599 3583 0.111841 0 - 0.36018 2024-04-22T06:49:06.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2003 16 3599 3583 0.111785 0 - 0.36018 2024-04-22T06:49:07.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2004 16 3599 3583 0.111729 0 - 0.36018 2024-04-22T06:49:08.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2005 16 3599 3583 0.111673 0 - 0.36018 2024-04-22T06:49:09.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2006 16 3599 3583 0.111618 0 - 0.36018 2024-04-22T06:49:10.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2007 16 3599 3583 0.111562 0 - 0.36018 2024-04-22T06:49:11.735 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2008 16 3599 3583 0.111507 0 - 0.36018 2024-04-22T06:49:12.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2009 16 3599 3583 0.111451 0 - 0.36018 2024-04-22T06:49:13.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2010 16 3599 3583 0.111396 0 - 0.36018 2024-04-22T06:49:14.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2011 16 3599 3583 0.11134 0 - 0.36018 2024-04-22T06:49:15.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2012 16 3599 3583 0.111285 0 - 0.36018 2024-04-22T06:49:16.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2013 16 3599 3583 0.11123 0 - 0.36018 2024-04-22T06:49:17.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2014 16 3599 3583 0.111174 0 - 0.36018 2024-04-22T06:49:18.736 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2015 16 3599 3583 0.111119 0 - 0.36018 2024-04-22T06:49:19.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2016 16 3599 3583 0.111064 0 - 0.36018 2024-04-22T06:49:20.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2017 16 3599 3583 0.111009 0 - 0.36018 2024-04-22T06:49:21.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2018 16 3599 3583 0.110954 0 - 0.36018 2024-04-22T06:49:22.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2019 16 3599 3583 0.110899 0 - 0.36018 2024-04-22T06:49:23.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:49:23.740365+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:49:23.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:49:23.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2020 16 3599 3583 0.110844 0 - 0.36018 2024-04-22T06:49:24.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2021 16 3599 3583 0.110789 0 - 0.36018 2024-04-22T06:49:25.737 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2022 16 3599 3583 0.110735 0 - 0.36018 2024-04-22T06:49:26.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2023 16 3599 3583 0.11068 0 - 0.36018 2024-04-22T06:49:27.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2024 16 3599 3583 0.110625 0 - 0.36018 2024-04-22T06:49:28.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2025 16 3599 3583 0.110571 0 - 0.36018 2024-04-22T06:49:29.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2026 16 3599 3583 0.110516 0 - 0.36018 2024-04-22T06:49:30.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2027 16 3599 3583 0.110461 0 - 0.36018 2024-04-22T06:49:31.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2028 16 3599 3583 0.110407 0 - 0.36018 2024-04-22T06:49:32.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2029 16 3599 3583 0.110353 0 - 0.36018 2024-04-22T06:49:33.738 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2030 16 3599 3583 0.110298 0 - 0.36018 2024-04-22T06:49:34.739 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2031 16 3599 3583 0.110244 0 - 0.36018 2024-04-22T06:49:35.739 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2032 16 3599 3583 0.11019 0 - 0.36018 2024-04-22T06:49:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:49:36.383 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:49:36.739 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2033 16 3599 3583 0.110135 0 - 0.36018 2024-04-22T06:49:37.739 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2034 16 3599 3583 0.110081 0 - 0.36018 2024-04-22T06:49:38.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:49:38.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:49:38.508+0000 7f1b30a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:49:38.739 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2035 16 3599 3583 0.110027 0 - 0.36018 2024-04-22T06:49:39.739 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2036 16 3599 3583 0.109973 0 - 0.36018 2024-04-22T06:49:40.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2037 16 3599 3583 0.109919 0 - 0.36018 2024-04-22T06:49:41.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2038 16 3599 3583 0.109865 0 - 0.36018 2024-04-22T06:49:42.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2039 16 3599 3583 0.109811 0 - 0.36018 2024-04-22T06:49:43.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:49:43.743285+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:49:43.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:49:43.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2040 16 3599 3583 0.109757 0 - 0.36018 2024-04-22T06:49:44.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2041 16 3599 3583 0.109704 0 - 0.36018 2024-04-22T06:49:45.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2042 16 3599 3583 0.10965 0 - 0.36018 2024-04-22T06:49:46.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2043 16 3599 3583 0.109596 0 - 0.36018 2024-04-22T06:49:47.740 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2044 16 3599 3583 0.109543 0 - 0.36018 2024-04-22T06:49:48.741 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2045 16 3599 3583 0.109489 0 - 0.36018 2024-04-22T06:49:49.741 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2046 16 3599 3583 0.109436 0 - 0.36018 2024-04-22T06:49:50.741 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2047 16 3599 3583 0.109382 0 - 0.36018 2024-04-22T06:49:51.741 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2048 16 3599 3583 0.109329 0 - 0.36018 2024-04-22T06:49:52.741 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2049 16 3599 3583 0.109275 0 - 0.36018 2024-04-22T06:49:53.741 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2050 16 3599 3583 0.109222 0 - 0.36018 2024-04-22T06:49:54.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2051 16 3599 3583 0.109169 0 - 0.36018 2024-04-22T06:49:55.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2052 16 3599 3583 0.109116 0 - 0.36018 2024-04-22T06:49:56.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2053 16 3599 3583 0.109062 0 - 0.36018 2024-04-22T06:49:57.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2054 16 3599 3583 0.109009 0 - 0.36018 2024-04-22T06:49:58.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2055 16 3599 3583 0.108956 0 - 0.36018 2024-04-22T06:49:59.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2056 16 3599 3583 0.108903 0 - 0.36018 2024-04-22T06:50:00.742 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2057 16 3599 3583 0.10885 0 - 0.36018 2024-04-22T06:50:01.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2058 16 3599 3583 0.108798 0 - 0.36018 2024-04-22T06:50:02.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2059 16 3599 3583 0.108745 0 - 0.36018 2024-04-22T06:50:03.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:50:03.746045+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:50:03.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:50:03.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2060 16 3599 3583 0.108692 0 - 0.36018 2024-04-22T06:50:04.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2061 16 3599 3583 0.108639 0 - 0.36018 2024-04-22T06:50:05.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2062 16 3599 3583 0.108586 0 - 0.36018 2024-04-22T06:50:06.743 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2063 16 3599 3583 0.108534 0 - 0.36018 2024-04-22T06:50:07.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2064 16 3599 3583 0.108481 0 - 0.36018 2024-04-22T06:50:08.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2065 16 3599 3583 0.108429 0 - 0.36018 2024-04-22T06:50:09.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2066 16 3599 3583 0.108376 0 - 0.36018 2024-04-22T06:50:10.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2067 16 3599 3583 0.108324 0 - 0.36018 2024-04-22T06:50:11.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2068 16 3599 3583 0.108271 0 - 0.36018 2024-04-22T06:50:12.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2069 16 3599 3583 0.108219 0 - 0.36018 2024-04-22T06:50:13.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2070 16 3599 3583 0.108167 0 - 0.36018 2024-04-22T06:50:14.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2071 16 3599 3583 0.108115 0 - 0.36018 2024-04-22T06:50:15.744 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2072 16 3599 3583 0.108062 0 - 0.36018 2024-04-22T06:50:16.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2073 16 3599 3583 0.10801 0 - 0.36018 2024-04-22T06:50:17.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2074 16 3599 3583 0.107958 0 - 0.36018 2024-04-22T06:50:18.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2075 16 3599 3583 0.107906 0 - 0.36018 2024-04-22T06:50:19.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2076 16 3599 3583 0.107854 0 - 0.36018 2024-04-22T06:50:20.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2077 16 3599 3583 0.107802 0 - 0.36018 2024-04-22T06:50:21.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2078 16 3599 3583 0.10775 0 - 0.36018 2024-04-22T06:50:22.745 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2079 16 3599 3583 0.107699 0 - 0.36018 2024-04-22T06:50:23.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:50:23.748870+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:50:23.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:50:23.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2080 16 3599 3583 0.107647 0 - 0.36018 2024-04-22T06:50:24.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2081 16 3599 3583 0.107595 0 - 0.36018 2024-04-22T06:50:25.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2082 16 3599 3583 0.107543 0 - 0.36018 2024-04-22T06:50:26.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2083 16 3599 3583 0.107492 0 - 0.36018 2024-04-22T06:50:27.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2084 16 3599 3583 0.10744 0 - 0.36018 2024-04-22T06:50:28.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2085 16 3599 3583 0.107389 0 - 0.36018 2024-04-22T06:50:29.746 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2086 16 3599 3583 0.107337 0 - 0.36018 2024-04-22T06:50:30.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2087 16 3599 3583 0.107286 0 - 0.36018 2024-04-22T06:50:31.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2088 16 3599 3583 0.107234 0 - 0.36018 2024-04-22T06:50:32.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2089 16 3599 3583 0.107183 0 - 0.36018 2024-04-22T06:50:33.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2090 16 3599 3583 0.107132 0 - 0.36018 2024-04-22T06:50:34.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2091 16 3599 3583 0.10708 0 - 0.36018 2024-04-22T06:50:35.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2092 16 3599 3583 0.107029 0 - 0.36018 2024-04-22T06:50:36.747 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2093 16 3599 3583 0.106978 0 - 0.36018 2024-04-22T06:50:37.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2094 16 3599 3583 0.106927 0 - 0.36018 2024-04-22T06:50:38.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2095 16 3599 3583 0.106876 0 - 0.36018 2024-04-22T06:50:39.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2096 16 3599 3583 0.106825 0 - 0.36018 2024-04-22T06:50:40.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2097 16 3599 3583 0.106774 0 - 0.36018 2024-04-22T06:50:41.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2098 16 3599 3583 0.106723 0 - 0.36018 2024-04-22T06:50:42.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2099 16 3599 3583 0.106672 0 - 0.36018 2024-04-22T06:50:43.748 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:50:43.751689+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:50:43.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:50:43.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2100 16 3599 3583 0.106622 0 - 0.36018 2024-04-22T06:50:44.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2101 16 3599 3583 0.106571 0 - 0.36018 2024-04-22T06:50:45.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2102 16 3599 3583 0.10652 0 - 0.36018 2024-04-22T06:50:46.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2103 16 3599 3583 0.106469 0 - 0.36018 2024-04-22T06:50:47.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2104 16 3599 3583 0.106419 0 - 0.36018 2024-04-22T06:50:48.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2105 16 3599 3583 0.106368 0 - 0.36018 2024-04-22T06:50:49.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2106 16 3599 3583 0.106318 0 - 0.36018 2024-04-22T06:50:50.749 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2107 16 3599 3583 0.106267 0 - 0.36018 2024-04-22T06:50:51.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2108 16 3599 3583 0.106217 0 - 0.36018 2024-04-22T06:50:52.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2109 16 3599 3583 0.106167 0 - 0.36018 2024-04-22T06:50:53.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2110 16 3599 3583 0.106116 0 - 0.36018 2024-04-22T06:50:54.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2111 16 3599 3583 0.106066 0 - 0.36018 2024-04-22T06:50:55.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2112 16 3599 3583 0.106016 0 - 0.36018 2024-04-22T06:50:56.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2113 16 3599 3583 0.105966 0 - 0.36018 2024-04-22T06:50:57.750 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2114 16 3599 3583 0.105915 0 - 0.36018 2024-04-22T06:50:58.751 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2115 16 3599 3583 0.105865 0 - 0.36018 2024-04-22T06:50:59.751 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2116 16 3599 3583 0.105815 0 - 0.36018 2024-04-22T06:51:00.751 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2117 16 3599 3583 0.105765 0 - 0.36018 2024-04-22T06:51:01.751 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2118 16 3599 3583 0.105715 0 - 0.36018 2024-04-22T06:51:02.751 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2119 16 3599 3583 0.105666 0 - 0.36018 2024-04-22T06:51:03.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:51:03.754629+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:51:03.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:51:03.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2120 16 3599 3583 0.105616 0 - 0.36018 2024-04-22T06:51:04.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2121 16 3599 3583 0.105566 0 - 0.36018 2024-04-22T06:51:05.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2122 16 3599 3583 0.105516 0 - 0.36018 2024-04-22T06:51:06.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2123 16 3599 3583 0.105466 0 - 0.36018 2024-04-22T06:51:07.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2124 16 3599 3583 0.105417 0 - 0.36018 2024-04-22T06:51:08.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2125 16 3599 3583 0.105367 0 - 0.36018 2024-04-22T06:51:09.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2126 16 3599 3583 0.105318 0 - 0.36018 2024-04-22T06:51:10.752 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2127 16 3599 3583 0.105268 0 - 0.36018 2024-04-22T06:51:11.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2128 16 3599 3583 0.105219 0 - 0.36018 2024-04-22T06:51:12.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2129 16 3599 3583 0.105169 0 - 0.36018 2024-04-22T06:51:13.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2130 16 3599 3583 0.10512 0 - 0.36018 2024-04-22T06:51:14.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2131 16 3599 3583 0.105071 0 - 0.36018 2024-04-22T06:51:15.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2132 16 3599 3583 0.105021 0 - 0.36018 2024-04-22T06:51:16.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2133 16 3599 3583 0.104972 0 - 0.36018 2024-04-22T06:51:17.753 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2134 16 3599 3583 0.104923 0 - 0.36018 2024-04-22T06:51:18.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2135 16 3599 3583 0.104874 0 - 0.36018 2024-04-22T06:51:19.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2136 16 3599 3583 0.104825 0 - 0.36018 2024-04-22T06:51:20.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2137 16 3599 3583 0.104776 0 - 0.36018 2024-04-22T06:51:21.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2138 16 3599 3583 0.104727 0 - 0.36018 2024-04-22T06:51:22.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2139 16 3599 3583 0.104678 0 - 0.36018 2024-04-22T06:51:23.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:51:23.757448+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:51:23.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:51:23.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2140 16 3599 3583 0.104629 0 - 0.36018 2024-04-22T06:51:24.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2141 16 3599 3583 0.10458 0 - 0.36018 2024-04-22T06:51:25.754 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2142 16 3599 3583 0.104531 0 - 0.36018 2024-04-22T06:51:26.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2143 16 3599 3583 0.104482 0 - 0.36018 2024-04-22T06:51:27.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2144 16 3599 3583 0.104433 0 - 0.36018 2024-04-22T06:51:28.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2145 16 3599 3583 0.104385 0 - 0.36018 2024-04-22T06:51:29.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2146 16 3599 3583 0.104336 0 - 0.36018 2024-04-22T06:51:30.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2147 16 3599 3583 0.104288 0 - 0.36018 2024-04-22T06:51:31.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2148 16 3599 3583 0.104239 0 - 0.36018 2024-04-22T06:51:32.755 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2149 16 3599 3583 0.10419 0 - 0.36018 2024-04-22T06:51:33.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2150 16 3599 3583 0.104142 0 - 0.36018 2024-04-22T06:51:34.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2151 16 3599 3583 0.104094 0 - 0.36018 2024-04-22T06:51:35.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2152 16 3599 3583 0.104045 0 - 0.36018 2024-04-22T06:51:36.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2153 16 3599 3583 0.103997 0 - 0.36018 2024-04-22T06:51:37.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2154 16 3599 3583 0.103949 0 - 0.36018 2024-04-22T06:51:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:51:38.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:51:38.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2155 16 3599 3583 0.1039 0 - 0.36018 2024-04-22T06:51:39.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2156 16 3599 3583 0.103852 0 - 0.36018 2024-04-22T06:51:40.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:51:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:51:40.548+0000 7f777366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:51:40.756 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2157 16 3599 3583 0.103804 0 - 0.36018 2024-04-22T06:51:41.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2158 16 3599 3583 0.103756 0 - 0.36018 2024-04-22T06:51:42.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2159 16 3599 3583 0.103708 0 - 0.36018 2024-04-22T06:51:43.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:51:43.760223+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:51:43.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:51:43.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2160 16 3599 3583 0.10366 0 - 0.36018 2024-04-22T06:51:44.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2161 16 3599 3583 0.103612 0 - 0.36018 2024-04-22T06:51:45.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2162 16 3599 3583 0.103564 0 - 0.36018 2024-04-22T06:51:46.757 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2163 16 3599 3583 0.103516 0 - 0.36018 2024-04-22T06:51:47.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2164 16 3599 3583 0.103468 0 - 0.36018 2024-04-22T06:51:48.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2165 16 3599 3583 0.10342 0 - 0.36018 2024-04-22T06:51:49.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2166 16 3599 3583 0.103373 0 - 0.36018 2024-04-22T06:51:50.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2167 16 3599 3583 0.103325 0 - 0.36018 2024-04-22T06:51:51.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2168 16 3599 3583 0.103277 0 - 0.36018 2024-04-22T06:51:52.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2169 16 3599 3583 0.10323 0 - 0.36018 2024-04-22T06:51:53.758 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2170 16 3599 3583 0.103182 0 - 0.36018 2024-04-22T06:51:54.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2171 16 3599 3583 0.103135 0 - 0.36018 2024-04-22T06:51:55.759 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2172 16 3599 3583 0.103087 0 - 0.36018 2024-04-22T06:51:56.759 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2173 16 3599 3583 0.10304 0 - 0.36018 2024-04-22T06:51:57.759 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2174 16 3599 3583 0.102992 0 - 0.36018 2024-04-22T06:51:58.759 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2175 16 3599 3583 0.102945 0 - 0.36018 2024-04-22T06:51:59.759 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2176 16 3599 3583 0.102898 0 - 0.36018 2024-04-22T06:52:00.759 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2177 16 3599 3583 0.10285 0 - 0.36018 2024-04-22T06:52:01.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2178 16 3599 3583 0.102803 0 - 0.36018 2024-04-22T06:52:02.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2179 16 3599 3583 0.102756 0 - 0.36018 2024-04-22T06:52:03.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:52:03.763059+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:52:03.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:52:03.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2180 16 3599 3583 0.102709 0 - 0.36018 2024-04-22T06:52:04.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2181 16 3599 3583 0.102662 0 - 0.36018 2024-04-22T06:52:05.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2182 16 3599 3583 0.102615 0 - 0.36018 2024-04-22T06:52:06.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2183 16 3599 3583 0.102568 0 - 0.36018 2024-04-22T06:52:07.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2184 16 3599 3583 0.102521 0 - 0.36018 2024-04-22T06:52:08.760 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2185 16 3599 3583 0.102474 0 - 0.36018 2024-04-22T06:52:09.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2186 16 3599 3583 0.102427 0 - 0.36018 2024-04-22T06:52:10.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2187 16 3599 3583 0.10238 0 - 0.36018 2024-04-22T06:52:11.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2188 16 3599 3583 0.102333 0 - 0.36018 2024-04-22T06:52:12.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2189 16 3599 3583 0.102287 0 - 0.36018 2024-04-22T06:52:13.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2190 16 3599 3583 0.10224 0 - 0.36018 2024-04-22T06:52:14.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2191 16 3599 3583 0.102193 0 - 0.36018 2024-04-22T06:52:15.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2192 16 3599 3583 0.102147 0 - 0.36018 2024-04-22T06:52:16.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2193 16 3599 3583 0.1021 0 - 0.36018 2024-04-22T06:52:17.761 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2194 16 3599 3583 0.102053 0 - 0.36018 2024-04-22T06:52:18.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2195 16 3599 3583 0.102007 0 - 0.36018 2024-04-22T06:52:19.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2196 16 3599 3583 0.101961 0 - 0.36018 2024-04-22T06:52:20.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2197 16 3599 3583 0.101914 0 - 0.36018 2024-04-22T06:52:21.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2198 16 3599 3583 0.101868 0 - 0.36018 2024-04-22T06:52:22.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2199 16 3599 3583 0.101821 0 - 0.36018 2024-04-22T06:52:23.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:52:23.765614+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:52:23.762 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:52:23.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2200 16 3599 3583 0.101775 0 - 0.36018 2024-04-22T06:52:24.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2201 16 3599 3583 0.101729 0 - 0.36018 2024-04-22T06:52:25.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2202 16 3599 3583 0.101683 0 - 0.36018 2024-04-22T06:52:26.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2203 16 3599 3583 0.101637 0 - 0.36018 2024-04-22T06:52:27.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2204 16 3599 3583 0.10159 0 - 0.36018 2024-04-22T06:52:28.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2205 16 3599 3583 0.101544 0 - 0.36018 2024-04-22T06:52:29.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2206 16 3599 3583 0.101498 0 - 0.36018 2024-04-22T06:52:30.763 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2207 16 3599 3583 0.101452 0 - 0.36018 2024-04-22T06:52:31.764 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2208 16 3599 3583 0.101406 0 - 0.36018 2024-04-22T06:52:32.764 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2209 16 3599 3583 0.10136 0 - 0.36018 2024-04-22T06:52:33.764 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2210 16 3599 3583 0.101315 0 - 0.36018 2024-04-22T06:52:34.764 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2211 16 3599 3583 0.101269 0 - 0.36018 2024-04-22T06:52:35.764 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2212 16 3599 3583 0.101223 0 - 0.36018 2024-04-22T06:52:36.764 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2213 16 3599 3583 0.101177 0 - 0.36018 2024-04-22T06:52:37.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2214 16 3599 3583 0.101132 0 - 0.36018 2024-04-22T06:52:38.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2215 16 3599 3583 0.101086 0 - 0.36018 2024-04-22T06:52:39.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2216 16 3599 3583 0.10104 0 - 0.36018 2024-04-22T06:52:40.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2217 16 3599 3583 0.100995 0 - 0.36018 2024-04-22T06:52:41.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2218 16 3599 3583 0.100949 0 - 0.36018 2024-04-22T06:52:42.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2219 16 3599 3583 0.100904 0 - 0.36018 2024-04-22T06:52:43.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:52:43.768485+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:52:43.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:52:43.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2220 16 3599 3583 0.100858 0 - 0.36018 2024-04-22T06:52:44.765 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2221 16 3599 3583 0.100813 0 - 0.36018 2024-04-22T06:52:45.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2222 16 3599 3583 0.100767 0 - 0.36018 2024-04-22T06:52:46.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2223 16 3599 3583 0.100722 0 - 0.36018 2024-04-22T06:52:47.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2224 16 3599 3583 0.100677 0 - 0.36018 2024-04-22T06:52:48.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2225 16 3599 3583 0.100632 0 - 0.36018 2024-04-22T06:52:49.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2226 16 3599 3583 0.100586 0 - 0.36018 2024-04-22T06:52:50.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2227 16 3599 3583 0.100541 0 - 0.36018 2024-04-22T06:52:51.766 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2228 16 3599 3583 0.100496 0 - 0.36018 2024-04-22T06:52:52.767 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2229 16 3599 3583 0.100451 0 - 0.36018 2024-04-22T06:52:53.767 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2230 16 3599 3583 0.100406 0 - 0.36018 2024-04-22T06:52:54.767 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2231 16 3599 3583 0.100361 0 - 0.36018 2024-04-22T06:52:55.767 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2232 16 3599 3583 0.100316 0 - 0.36018 2024-04-22T06:52:56.767 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2233 16 3599 3583 0.100271 0 - 0.36018 2024-04-22T06:52:57.767 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2234 16 3599 3583 0.100226 0 - 0.36018 2024-04-22T06:52:58.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2235 16 3599 3583 0.100181 0 - 0.36018 2024-04-22T06:52:59.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2236 16 3599 3583 0.100137 0 - 0.36018 2024-04-22T06:53:00.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2237 16 3599 3583 0.100092 0 - 0.36018 2024-04-22T06:53:01.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2238 16 3599 3583 0.100047 0 - 0.36018 2024-04-22T06:53:02.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2239 16 3599 3583 0.100002 0 - 0.36018 2024-04-22T06:53:03.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:53:03.771437+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:53:03.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:53:03.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2240 16 3599 3583 0.0999577 0 - 0.36018 2024-04-22T06:53:04.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2241 16 3599 3583 0.0999131 0 - 0.36018 2024-04-22T06:53:05.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2242 16 3599 3583 0.0998686 0 - 0.36018 2024-04-22T06:53:06.768 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2243 16 3599 3583 0.099824 0 - 0.36018 2024-04-22T06:53:07.769 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2244 16 3599 3583 0.0997796 0 - 0.36018 2024-04-22T06:53:08.769 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2245 16 3599 3583 0.0997351 0 - 0.36018 2024-04-22T06:53:09.769 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2246 16 3599 3583 0.0996907 0 - 0.36018 2024-04-22T06:53:10.769 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2247 16 3599 3583 0.0996463 0 - 0.36018 2024-04-22T06:53:11.769 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2248 16 3599 3583 0.099602 0 - 0.36018 2024-04-22T06:53:12.769 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2249 16 3599 3583 0.0995577 0 - 0.36018 2024-04-22T06:53:13.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2250 16 3599 3583 0.0995135 0 - 0.36018 2024-04-22T06:53:14.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2251 16 3599 3583 0.0994693 0 - 0.36018 2024-04-22T06:53:15.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2252 16 3599 3583 0.0994251 0 - 0.36018 2024-04-22T06:53:16.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2253 16 3599 3583 0.099381 0 - 0.36018 2024-04-22T06:53:17.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2254 16 3599 3583 0.0993369 0 - 0.36018 2024-04-22T06:53:18.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2255 16 3599 3583 0.0992928 0 - 0.36018 2024-04-22T06:53:19.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2256 16 3599 3583 0.0992488 0 - 0.36018 2024-04-22T06:53:20.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2257 16 3599 3583 0.0992049 0 - 0.36018 2024-04-22T06:53:21.770 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2258 16 3599 3583 0.0991609 0 - 0.36018 2024-04-22T06:53:22.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2259 16 3599 3583 0.099117 0 - 0.36018 2024-04-22T06:53:23.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:53:23.774068+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:53:23.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:53:23.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2260 16 3599 3583 0.0990732 0 - 0.36018 2024-04-22T06:53:24.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2261 16 3599 3583 0.0990293 0 - 0.36018 2024-04-22T06:53:25.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2262 16 3599 3583 0.0989856 0 - 0.36018 2024-04-22T06:53:26.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2263 16 3599 3583 0.0989418 0 - 0.36018 2024-04-22T06:53:27.771 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2264 16 3599 3583 0.0988981 0 - 0.36018 2024-04-22T06:53:28.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2265 16 3599 3583 0.0988545 0 - 0.36018 2024-04-22T06:53:29.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2266 16 3599 3583 0.0988108 0 - 0.36018 2024-04-22T06:53:30.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2267 16 3599 3583 0.0987673 0 - 0.36018 2024-04-22T06:53:31.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2268 16 3599 3583 0.0987237 0 - 0.36018 2024-04-22T06:53:32.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2269 16 3599 3583 0.0986802 0 - 0.36018 2024-04-22T06:53:33.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2270 16 3599 3583 0.0986367 0 - 0.36018 2024-04-22T06:53:34.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2271 16 3599 3583 0.0985933 0 - 0.36018 2024-04-22T06:53:35.772 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2272 16 3599 3583 0.0985499 0 - 0.36018 2024-04-22T06:53:36.773 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2273 16 3599 3583 0.0985065 0 - 0.36018 2024-04-22T06:53:37.773 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2274 16 3599 3583 0.0984632 0 - 0.36018 2024-04-22T06:53:38.773 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2275 16 3599 3583 0.0984199 0 - 0.36018 2024-04-22T06:53:39.773 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2276 16 3599 3583 0.0983767 0 - 0.36018 2024-04-22T06:53:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:53:40.459 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:53:40.773 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2277 16 3599 3583 0.0983335 0 - 0.36018 2024-04-22T06:53:41.773 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2278 16 3599 3583 0.0982903 0 - 0.36018 2024-04-22T06:53:42.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:53:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:53:42.583+0000 7f9a8742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:53:42.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2279 16 3599 3583 0.0982472 0 - 0.36018 2024-04-22T06:53:43.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:53:43.777024+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:53:43.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:53:43.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2280 16 3599 3583 0.0982041 0 - 0.36018 2024-04-22T06:53:44.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2281 16 3599 3583 0.098161 0 - 0.36018 2024-04-22T06:53:45.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2282 16 3599 3583 0.098118 0 - 0.36018 2024-04-22T06:53:46.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2283 16 3599 3583 0.0980751 0 - 0.36018 2024-04-22T06:53:47.774 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2284 16 3599 3583 0.0980321 0 - 0.36018 2024-04-22T06:53:48.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2285 16 3599 3583 0.0979892 0 - 0.36018 2024-04-22T06:53:49.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2286 16 3599 3583 0.0979463 0 - 0.36018 2024-04-22T06:53:50.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2287 16 3599 3583 0.0979035 0 - 0.36018 2024-04-22T06:53:51.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2288 16 3599 3583 0.0978607 0 - 0.36018 2024-04-22T06:53:52.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2289 16 3599 3583 0.097818 0 - 0.36018 2024-04-22T06:53:53.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2290 16 3599 3583 0.0977753 0 - 0.36018 2024-04-22T06:53:54.775 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2291 16 3599 3583 0.0977326 0 - 0.36018 2024-04-22T06:53:55.776 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2292 16 3599 3583 0.0976899 0 - 0.36018 2024-04-22T06:53:56.776 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2293 16 3599 3583 0.0976473 0 - 0.36018 2024-04-22T06:53:57.776 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2294 16 3599 3583 0.0976048 0 - 0.36018 2024-04-22T06:53:58.776 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2295 16 3599 3583 0.0975622 0 - 0.36018 2024-04-22T06:53:59.776 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2296 16 3599 3583 0.0975197 0 - 0.36018 2024-04-22T06:54:00.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2297 16 3599 3583 0.0974773 0 - 0.36018 2024-04-22T06:54:01.776 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2298 16 3599 3583 0.0974349 0 - 0.36018 2024-04-22T06:54:02.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2299 16 3599 3583 0.0973925 0 - 0.36018 2024-04-22T06:54:03.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:54:03.780008+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:54:03.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:54:03.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2300 16 3599 3583 0.0973501 0 - 0.36018 2024-04-22T06:54:04.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2301 16 3599 3583 0.0973078 0 - 0.36018 2024-04-22T06:54:05.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2302 16 3599 3583 0.0972656 0 - 0.36018 2024-04-22T06:54:06.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2303 16 3599 3583 0.0972233 0 - 0.36018 2024-04-22T06:54:07.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2304 16 3599 3583 0.0971811 0 - 0.36018 2024-04-22T06:54:08.777 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2305 16 3599 3583 0.097139 0 - 0.36018 2024-04-22T06:54:09.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2306 16 3599 3583 0.0970969 0 - 0.36018 2024-04-22T06:54:10.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2307 16 3599 3583 0.0970548 0 - 0.36018 2024-04-22T06:54:11.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2308 16 3599 3583 0.0970127 0 - 0.36018 2024-04-22T06:54:12.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2309 16 3599 3583 0.0969707 0 - 0.36018 2024-04-22T06:54:13.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2310 16 3599 3583 0.0969287 0 - 0.36018 2024-04-22T06:54:14.778 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2311 16 3599 3583 0.0968868 0 - 0.36018 2024-04-22T06:54:15.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2312 16 3599 3583 0.0968449 0 - 0.36018 2024-04-22T06:54:16.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2313 16 3599 3583 0.096803 0 - 0.36018 2024-04-22T06:54:17.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2314 16 3599 3583 0.0967612 0 - 0.36018 2024-04-22T06:54:18.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2315 16 3599 3583 0.0967194 0 - 0.36018 2024-04-22T06:54:19.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2316 16 3599 3583 0.0966776 0 - 0.36018 2024-04-22T06:54:20.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2317 16 3599 3583 0.0966359 0 - 0.36018 2024-04-22T06:54:21.779 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2318 16 3599 3583 0.0965942 0 - 0.36018 2024-04-22T06:54:22.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2319 16 3599 3583 0.0965525 0 - 0.36018 2024-04-22T06:54:23.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:54:23.782971+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:54:23.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:54:23.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2320 16 3599 3583 0.0965109 0 - 0.36018 2024-04-22T06:54:24.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2321 16 3599 3583 0.0964693 0 - 0.36018 2024-04-22T06:54:25.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2322 16 3599 3583 0.0964278 0 - 0.36018 2024-04-22T06:54:26.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2323 16 3599 3583 0.0963863 0 - 0.36018 2024-04-22T06:54:27.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2324 16 3599 3583 0.0963448 0 - 0.36018 2024-04-22T06:54:28.780 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2325 16 3599 3583 0.0963034 0 - 0.36018 2024-04-22T06:54:29.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2326 16 3599 3583 0.096262 0 - 0.36018 2024-04-22T06:54:30.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2327 16 3599 3583 0.0962206 0 - 0.36018 2024-04-22T06:54:31.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2328 16 3599 3583 0.0961793 0 - 0.36018 2024-04-22T06:54:32.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2329 16 3599 3583 0.096138 0 - 0.36018 2024-04-22T06:54:33.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2330 16 3599 3583 0.0960967 0 - 0.36018 2024-04-22T06:54:34.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2331 16 3599 3583 0.0960555 0 - 0.36018 2024-04-22T06:54:35.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2332 16 3599 3583 0.0960143 0 - 0.36018 2024-04-22T06:54:36.781 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2333 16 3599 3583 0.0959731 0 - 0.36018 2024-04-22T06:54:37.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2334 16 3599 3583 0.095932 0 - 0.36018 2024-04-22T06:54:38.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2335 16 3599 3583 0.0958909 0 - 0.36018 2024-04-22T06:54:39.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2336 16 3599 3583 0.0958499 0 - 0.36018 2024-04-22T06:54:40.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2337 16 3599 3583 0.0958089 0 - 0.36018 2024-04-22T06:54:41.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2338 16 3599 3583 0.0957679 0 - 0.36018 2024-04-22T06:54:42.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2339 16 3599 3583 0.095727 0 - 0.36018 2024-04-22T06:54:43.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:54:43.785662+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:54:43.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:54:43.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2340 16 3599 3583 0.095686 0 - 0.36018 2024-04-22T06:54:44.782 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2341 16 3599 3583 0.0956452 0 - 0.36018 2024-04-22T06:54:45.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2342 16 3599 3583 0.0956043 0 - 0.36018 2024-04-22T06:54:46.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2343 16 3599 3583 0.0955635 0 - 0.36018 2024-04-22T06:54:47.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2344 16 3599 3583 0.0955228 0 - 0.36018 2024-04-22T06:54:48.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2345 16 3599 3583 0.095482 0 - 0.36018 2024-04-22T06:54:49.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2346 16 3599 3583 0.0954413 0 - 0.36018 2024-04-22T06:54:50.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2347 16 3599 3583 0.0954007 0 - 0.36018 2024-04-22T06:54:51.783 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2348 16 3599 3583 0.09536 0 - 0.36018 2024-04-22T06:54:52.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2349 16 3599 3583 0.0953194 0 - 0.36018 2024-04-22T06:54:53.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2350 16 3599 3583 0.0952789 0 - 0.36018 2024-04-22T06:54:54.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2351 16 3599 3583 0.0952383 0 - 0.36018 2024-04-22T06:54:55.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2352 16 3599 3583 0.0951979 0 - 0.36018 2024-04-22T06:54:56.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2353 16 3599 3583 0.0951574 0 - 0.36018 2024-04-22T06:54:57.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2354 16 3599 3583 0.095117 0 - 0.36018 2024-04-22T06:54:58.784 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2355 16 3599 3583 0.0950766 0 - 0.36018 2024-04-22T06:54:59.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2356 16 3599 3583 0.0950362 0 - 0.36018 2024-04-22T06:55:00.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2357 16 3599 3583 0.0949959 0 - 0.36018 2024-04-22T06:55:01.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2358 16 3599 3583 0.0949556 0 - 0.36018 2024-04-22T06:55:02.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2359 16 3599 3583 0.0949154 0 - 0.36018 2024-04-22T06:55:03.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:55:03.788568+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:55:03.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:55:03.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2360 16 3599 3583 0.0948751 0 - 0.36018 2024-04-22T06:55:04.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2361 16 3599 3583 0.094835 0 - 0.36018 2024-04-22T06:55:05.785 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2362 16 3599 3583 0.0947948 0 - 0.36018 2024-04-22T06:55:06.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2363 16 3599 3583 0.0947547 0 - 0.36018 2024-04-22T06:55:07.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2364 16 3599 3583 0.0947146 0 - 0.36018 2024-04-22T06:55:08.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2365 16 3599 3583 0.0946746 0 - 0.36018 2024-04-22T06:55:09.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2366 16 3599 3583 0.0946346 0 - 0.36018 2024-04-22T06:55:10.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2367 16 3599 3583 0.0945946 0 - 0.36018 2024-04-22T06:55:11.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2368 16 3599 3583 0.0945546 0 - 0.36018 2024-04-22T06:55:12.786 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2369 16 3599 3583 0.0945147 0 - 0.36018 2024-04-22T06:55:13.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2370 16 3599 3583 0.0944748 0 - 0.36018 2024-04-22T06:55:14.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2371 16 3599 3583 0.094435 0 - 0.36018 2024-04-22T06:55:15.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2372 16 3599 3583 0.0943952 0 - 0.36018 2024-04-22T06:55:16.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2373 16 3599 3583 0.0943554 0 - 0.36018 2024-04-22T06:55:17.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2374 16 3599 3583 0.0943156 0 - 0.36018 2024-04-22T06:55:18.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2375 16 3599 3583 0.0942759 0 - 0.36018 2024-04-22T06:55:19.787 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2376 16 3599 3583 0.0942363 0 - 0.36018 2024-04-22T06:55:20.788 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2377 16 3599 3583 0.0941966 0 - 0.36018 2024-04-22T06:55:21.788 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2378 16 3599 3583 0.094157 0 - 0.36018 2024-04-22T06:55:22.788 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2379 16 3599 3583 0.0941174 0 - 0.36018 2024-04-22T06:55:23.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:55:23.791448+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:55:23.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:55:23.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2380 16 3599 3583 0.0940779 0 - 0.36018 2024-04-22T06:55:24.788 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2381 16 3599 3583 0.0940384 0 - 0.36018 2024-04-22T06:55:25.788 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2382 16 3599 3583 0.0939989 0 - 0.36018 2024-04-22T06:55:26.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2383 16 3599 3583 0.0939594 0 - 0.36018 2024-04-22T06:55:27.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2384 16 3599 3583 0.09392 0 - 0.36018 2024-04-22T06:55:28.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2385 16 3599 3583 0.0938806 0 - 0.36018 2024-04-22T06:55:29.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2386 16 3599 3583 0.0938413 0 - 0.36018 2024-04-22T06:55:30.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2387 16 3599 3583 0.093802 0 - 0.36018 2024-04-22T06:55:31.789 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2388 16 3599 3583 0.0937627 0 - 0.36018 2024-04-22T06:55:32.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2389 16 3599 3583 0.0937235 0 - 0.36018 2024-04-22T06:55:33.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2390 16 3599 3583 0.0936842 0 - 0.36018 2024-04-22T06:55:34.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2391 16 3599 3583 0.0936451 0 - 0.36018 2024-04-22T06:55:35.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2392 16 3599 3583 0.0936059 0 - 0.36018 2024-04-22T06:55:36.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2393 16 3599 3583 0.0935668 0 - 0.36018 2024-04-22T06:55:37.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2394 16 3599 3583 0.0935277 0 - 0.36018 2024-04-22T06:55:38.790 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2395 16 3599 3583 0.0934887 0 - 0.36018 2024-04-22T06:55:39.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2396 16 3599 3583 0.0934496 0 - 0.36018 2024-04-22T06:55:40.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2397 16 3599 3583 0.0934107 0 - 0.36018 2024-04-22T06:55:41.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2398 16 3599 3583 0.0933717 0 - 0.36018 2024-04-22T06:55:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:55:42.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:55:42.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2399 16 3599 3583 0.0933328 0 - 0.36018 2024-04-22T06:55:43.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:55:43.794368+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:55:43.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:55:43.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2400 16 3599 3583 0.0932939 0 - 0.36018 2024-04-22T06:55:44.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:55:44.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:55:44.620+0000 7fb3b935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:55:44.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2401 16 3599 3583 0.093255 0 - 0.36018 2024-04-22T06:55:45.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2402 16 3599 3583 0.0932162 0 - 0.36018 2024-04-22T06:55:46.791 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2403 16 3599 3583 0.0931774 0 - 0.36018 2024-04-22T06:55:47.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2404 16 3599 3583 0.0931387 0 - 0.36018 2024-04-22T06:55:48.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2405 16 3599 3583 0.0930999 0 - 0.36018 2024-04-22T06:55:49.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2406 16 3599 3583 0.0930612 0 - 0.36018 2024-04-22T06:55:50.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2407 16 3599 3583 0.0930226 0 - 0.36018 2024-04-22T06:55:51.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2408 16 3599 3583 0.092984 0 - 0.36018 2024-04-22T06:55:52.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2409 16 3599 3583 0.0929454 0 - 0.36018 2024-04-22T06:55:53.792 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2410 16 3599 3583 0.0929068 0 - 0.36018 2024-04-22T06:55:54.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2411 16 3599 3583 0.0928683 0 - 0.36018 2024-04-22T06:55:55.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2412 16 3599 3583 0.0928298 0 - 0.36018 2024-04-22T06:55:56.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2413 16 3599 3583 0.0927913 0 - 0.36018 2024-04-22T06:55:57.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2414 16 3599 3583 0.0927528 0 - 0.36018 2024-04-22T06:55:58.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2415 16 3599 3583 0.0927144 0 - 0.36018 2024-04-22T06:55:59.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2416 16 3599 3583 0.0926761 0 - 0.36018 2024-04-22T06:56:00.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2417 16 3599 3583 0.0926377 0 - 0.36018 2024-04-22T06:56:01.793 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2418 16 3599 3583 0.0925994 0 - 0.36018 2024-04-22T06:56:02.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2419 16 3599 3583 0.0925611 0 - 0.36018 2024-04-22T06:56:03.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:56:03.797062+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:56:03.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:56:03.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2420 16 3599 3583 0.0925229 0 - 0.36018 2024-04-22T06:56:04.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2421 16 3599 3583 0.0924847 0 - 0.36018 2024-04-22T06:56:05.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2422 16 3599 3583 0.0924465 0 - 0.36018 2024-04-22T06:56:06.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2423 16 3599 3583 0.0924083 0 - 0.36018 2024-04-22T06:56:07.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2424 16 3599 3583 0.0923702 0 - 0.36018 2024-04-22T06:56:08.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2425 16 3599 3583 0.0923321 0 - 0.36018 2024-04-22T06:56:09.794 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2426 16 3599 3583 0.092294 0 - 0.36018 2024-04-22T06:56:10.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2427 16 3599 3583 0.092256 0 - 0.36018 2024-04-22T06:56:11.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2428 16 3599 3583 0.092218 0 - 0.36018 2024-04-22T06:56:12.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2429 16 3599 3583 0.0921801 0 - 0.36018 2024-04-22T06:56:13.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2430 16 3599 3583 0.0921421 0 - 0.36018 2024-04-22T06:56:14.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2431 16 3599 3583 0.0921042 0 - 0.36018 2024-04-22T06:56:15.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2432 16 3599 3583 0.0920664 0 - 0.36018 2024-04-22T06:56:16.795 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2433 16 3599 3583 0.0920285 0 - 0.36018 2024-04-22T06:56:17.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2434 16 3599 3583 0.0919907 0 - 0.36018 2024-04-22T06:56:18.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2435 16 3599 3583 0.0919529 0 - 0.36018 2024-04-22T06:56:19.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2436 16 3599 3583 0.0919152 0 - 0.36018 2024-04-22T06:56:20.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2437 16 3599 3583 0.0918775 0 - 0.36018 2024-04-22T06:56:21.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2438 16 3599 3583 0.0918398 0 - 0.36018 2024-04-22T06:56:22.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2439 16 3599 3583 0.0918021 0 - 0.36018 2024-04-22T06:56:23.796 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:56:23.799860+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:56:23.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:56:23.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2440 16 3599 3583 0.0917645 0 - 0.36018 2024-04-22T06:56:24.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2441 16 3599 3583 0.0917269 0 - 0.36018 2024-04-22T06:56:25.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2442 16 3599 3583 0.0916893 0 - 0.36018 2024-04-22T06:56:26.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2443 16 3599 3583 0.0916518 0 - 0.36018 2024-04-22T06:56:27.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2444 16 3599 3583 0.0916143 0 - 0.36018 2024-04-22T06:56:28.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2445 16 3599 3583 0.0915768 0 - 0.36018 2024-04-22T06:56:29.797 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2446 16 3599 3583 0.0915394 0 - 0.36018 2024-04-22T06:56:30.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2447 16 3599 3583 0.091502 0 - 0.36018 2024-04-22T06:56:31.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2448 16 3599 3583 0.0914646 0 - 0.36018 2024-04-22T06:56:32.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2449 16 3599 3583 0.0914273 0 - 0.36018 2024-04-22T06:56:33.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2450 16 3599 3583 0.0913899 0 - 0.36018 2024-04-22T06:56:34.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2451 16 3599 3583 0.0913527 0 - 0.36018 2024-04-22T06:56:35.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2452 16 3599 3583 0.0913154 0 - 0.36018 2024-04-22T06:56:36.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2453 16 3599 3583 0.0912782 0 - 0.36018 2024-04-22T06:56:37.798 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2454 16 3599 3583 0.091241 0 - 0.36018 2024-04-22T06:56:38.799 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2455 16 3599 3583 0.0912038 0 - 0.36018 2024-04-22T06:56:39.799 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2456 16 3599 3583 0.0911667 0 - 0.36018 2024-04-22T06:56:40.799 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2457 16 3599 3583 0.0911296 0 - 0.36018 2024-04-22T06:56:41.799 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2458 16 3599 3583 0.0910925 0 - 0.36018 2024-04-22T06:56:42.799 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2459 16 3599 3583 0.0910555 0 - 0.36018 2024-04-22T06:56:43.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:56:43.802666+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:56:43.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:56:43.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2460 16 3599 3583 0.0910184 0 - 0.36018 2024-04-22T06:56:44.799 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2461 16 3599 3583 0.0909815 0 - 0.36018 2024-04-22T06:56:45.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2462 16 3599 3583 0.0909445 0 - 0.36018 2024-04-22T06:56:46.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2463 16 3599 3583 0.0909076 0 - 0.36018 2024-04-22T06:56:47.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2464 16 3599 3583 0.0908707 0 - 0.36018 2024-04-22T06:56:48.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2465 16 3599 3583 0.0908338 0 - 0.36018 2024-04-22T06:56:49.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2466 16 3599 3583 0.090797 0 - 0.36018 2024-04-22T06:56:50.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2467 16 3599 3583 0.0907602 0 - 0.36018 2024-04-22T06:56:51.800 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2468 16 3599 3583 0.0907234 0 - 0.36018 2024-04-22T06:56:52.801 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2469 16 3599 3583 0.0906867 0 - 0.36018 2024-04-22T06:56:53.801 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2470 16 3599 3583 0.0906499 0 - 0.36018 2024-04-22T06:56:54.801 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2471 16 3599 3583 0.0906133 0 - 0.36018 2024-04-22T06:56:55.801 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2472 16 3599 3583 0.0905766 0 - 0.36018 2024-04-22T06:56:56.801 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2473 16 3599 3583 0.09054 0 - 0.36018 2024-04-22T06:56:57.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2474 16 3599 3583 0.0905034 0 - 0.36018 2024-04-22T06:56:58.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2475 16 3599 3583 0.0904668 0 - 0.36018 2024-04-22T06:56:59.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2476 16 3599 3583 0.0904303 0 - 0.36018 2024-04-22T06:57:00.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2477 16 3599 3583 0.0903938 0 - 0.36018 2024-04-22T06:57:01.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2478 16 3599 3583 0.0903573 0 - 0.36018 2024-04-22T06:57:02.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2479 16 3599 3583 0.0903208 0 - 0.36018 2024-04-22T06:57:03.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:57:03.805524+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:57:03.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:57:03.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2480 16 3599 3583 0.0902844 0 - 0.36018 2024-04-22T06:57:04.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2481 16 3599 3583 0.090248 0 - 0.36018 2024-04-22T06:57:05.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2482 16 3599 3583 0.0902117 0 - 0.36018 2024-04-22T06:57:06.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2483 16 3599 3583 0.0901753 0 - 0.36018 2024-04-22T06:57:07.802 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2484 16 3599 3583 0.090139 0 - 0.36018 2024-04-22T06:57:08.803 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2485 16 3599 3583 0.0901028 0 - 0.36018 2024-04-22T06:57:09.803 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2486 16 3599 3583 0.0900665 0 - 0.36018 2024-04-22T06:57:10.803 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2487 16 3599 3583 0.0900303 0 - 0.36018 2024-04-22T06:57:11.803 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2488 16 3599 3583 0.0899941 0 - 0.36018 2024-04-22T06:57:12.803 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2489 16 3599 3583 0.089958 0 - 0.36018 2024-04-22T06:57:13.803 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2490 16 3599 3583 0.0899218 0 - 0.36018 2024-04-22T06:57:14.804 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2491 16 3599 3583 0.0898857 0 - 0.36018 2024-04-22T06:57:15.804 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2492 16 3599 3583 0.0898497 0 - 0.36018 2024-04-22T06:57:16.804 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2493 16 3599 3583 0.0898136 0 - 0.36018 2024-04-22T06:57:17.804 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2494 16 3599 3583 0.0897776 0 - 0.36018 2024-04-22T06:57:18.804 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2495 16 3599 3583 0.0897416 0 - 0.36018 2024-04-22T06:57:19.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2496 16 3599 3583 0.0897057 0 - 0.36018 2024-04-22T06:57:20.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2497 16 3599 3583 0.0896698 0 - 0.36018 2024-04-22T06:57:21.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2498 16 3599 3583 0.0896339 0 - 0.36018 2024-04-22T06:57:22.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2499 16 3599 3583 0.089598 0 - 0.36018 2024-04-22T06:57:23.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:57:23.808396+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:57:23.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:57:23.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2500 16 3599 3583 0.0895621 0 - 0.36018 2024-04-22T06:57:24.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2501 16 3599 3583 0.0895263 0 - 0.36018 2024-04-22T06:57:25.805 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2502 16 3599 3583 0.0894906 0 - 0.36018 2024-04-22T06:57:26.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2503 16 3599 3583 0.0894548 0 - 0.36018 2024-04-22T06:57:27.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2504 16 3599 3583 0.0894191 0 - 0.36018 2024-04-22T06:57:28.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2505 16 3599 3583 0.0893834 0 - 0.36018 2024-04-22T06:57:29.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2506 16 3599 3583 0.0893477 0 - 0.36018 2024-04-22T06:57:30.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2507 16 3599 3583 0.0893121 0 - 0.36018 2024-04-22T06:57:31.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2508 16 3599 3583 0.0892765 0 - 0.36018 2024-04-22T06:57:32.806 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2509 16 3599 3583 0.0892409 0 - 0.36018 2024-04-22T06:57:33.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2510 16 3599 3583 0.0892053 0 - 0.36018 2024-04-22T06:57:34.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2511 16 3599 3583 0.0891698 0 - 0.36018 2024-04-22T06:57:35.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2512 16 3599 3583 0.0891343 0 - 0.36018 2024-04-22T06:57:36.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2513 16 3599 3583 0.0890988 0 - 0.36018 2024-04-22T06:57:37.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2514 16 3599 3583 0.0890634 0 - 0.36018 2024-04-22T06:57:38.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2515 16 3599 3583 0.089028 0 - 0.36018 2024-04-22T06:57:39.807 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2516 16 3599 3583 0.0889926 0 - 0.36018 2024-04-22T06:57:40.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2517 16 3599 3583 0.0889572 0 - 0.36018 2024-04-22T06:57:41.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2518 16 3599 3583 0.0889219 0 - 0.36018 2024-04-22T06:57:42.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2519 16 3599 3583 0.0888866 0 - 0.36018 2024-04-22T06:57:43.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:57:43.811257+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:57:43.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:57:43.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2520 16 3599 3583 0.0888513 0 - 0.36018 2024-04-22T06:57:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:57:44.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:57:44.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2521 16 3599 3583 0.0888161 0 - 0.36018 2024-04-22T06:57:45.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2522 16 3599 3583 0.0887809 0 - 0.36018 2024-04-22T06:57:46.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T06:57:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:57:46.659+0000 7fcbc3d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:57:46.808 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2523 16 3599 3583 0.0887457 0 - 0.36018 2024-04-22T06:57:47.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2524 16 3599 3583 0.0887105 0 - 0.36018 2024-04-22T06:57:48.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2525 16 3599 3583 0.0886754 0 - 0.36018 2024-04-22T06:57:49.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2526 16 3599 3583 0.0886403 0 - 0.36018 2024-04-22T06:57:50.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2527 16 3599 3583 0.0886052 0 - 0.36018 2024-04-22T06:57:51.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2528 16 3599 3583 0.0885702 0 - 0.36018 2024-04-22T06:57:52.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2529 16 3599 3583 0.0885351 0 - 0.36018 2024-04-22T06:57:53.809 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2530 16 3599 3583 0.0885001 0 - 0.36018 2024-04-22T06:57:54.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2531 16 3599 3583 0.0884652 0 - 0.36018 2024-04-22T06:57:55.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2532 16 3599 3583 0.0884302 0 - 0.36018 2024-04-22T06:57:56.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2533 16 3599 3583 0.0883953 0 - 0.36018 2024-04-22T06:57:57.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2534 16 3599 3583 0.0883605 0 - 0.36018 2024-04-22T06:57:58.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2535 16 3599 3583 0.0883256 0 - 0.36018 2024-04-22T06:57:59.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2536 16 3599 3583 0.0882908 0 - 0.36018 2024-04-22T06:58:00.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2537 16 3599 3583 0.088256 0 - 0.36018 2024-04-22T06:58:01.810 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2538 16 3599 3583 0.0882212 0 - 0.36018 2024-04-22T06:58:02.811 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2539 16 3599 3583 0.0881864 0 - 0.36018 2024-04-22T06:58:03.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:58:03.814110+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:58:03.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:58:03.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2540 16 3599 3583 0.0881517 0 - 0.36018 2024-04-22T06:58:04.811 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2541 16 3599 3583 0.088117 0 - 0.36018 2024-04-22T06:58:05.811 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2542 16 3599 3583 0.0880824 0 - 0.36018 2024-04-22T06:58:06.811 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2543 16 3599 3583 0.0880477 0 - 0.36018 2024-04-22T06:58:07.811 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2544 16 3599 3583 0.0880131 0 - 0.36018 2024-04-22T06:58:08.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2545 16 3599 3583 0.0879785 0 - 0.36018 2024-04-22T06:58:09.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2546 16 3599 3583 0.087944 0 - 0.36018 2024-04-22T06:58:10.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2547 16 3599 3583 0.0879095 0 - 0.36018 2024-04-22T06:58:11.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2548 16 3599 3583 0.0878749 0 - 0.36018 2024-04-22T06:58:12.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2549 16 3599 3583 0.0878405 0 - 0.36018 2024-04-22T06:58:13.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2550 16 3599 3583 0.087806 0 - 0.36018 2024-04-22T06:58:14.812 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2551 16 3599 3583 0.0877716 0 - 0.36018 2024-04-22T06:58:15.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2552 16 3599 3583 0.0877372 0 - 0.36018 2024-04-22T06:58:16.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2553 16 3599 3583 0.0877028 0 - 0.36018 2024-04-22T06:58:17.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2554 16 3599 3583 0.0876685 0 - 0.36018 2024-04-22T06:58:18.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2555 16 3599 3583 0.0876342 0 - 0.36018 2024-04-22T06:58:19.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2556 16 3599 3583 0.0875999 0 - 0.36018 2024-04-22T06:58:20.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2557 16 3599 3583 0.0875657 0 - 0.36018 2024-04-22T06:58:21.813 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2558 16 3599 3583 0.0875314 0 - 0.36018 2024-04-22T06:58:22.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2559 16 3599 3583 0.0874972 0 - 0.36018 2024-04-22T06:58:23.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:58:23.817090+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:58:23.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:58:23.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2560 16 3599 3583 0.087463 0 - 0.36018 2024-04-22T06:58:24.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2561 16 3599 3583 0.0874289 0 - 0.36018 2024-04-22T06:58:25.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2562 16 3599 3583 0.0873948 0 - 0.36018 2024-04-22T06:58:26.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2563 16 3599 3583 0.0873607 0 - 0.36018 2024-04-22T06:58:27.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2564 16 3599 3583 0.0873266 0 - 0.36018 2024-04-22T06:58:28.814 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2565 16 3599 3583 0.0872925 0 - 0.36018 2024-04-22T06:58:29.815 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2566 16 3599 3583 0.0872585 0 - 0.36018 2024-04-22T06:58:30.815 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2567 16 3599 3583 0.0872245 0 - 0.36018 2024-04-22T06:58:31.815 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2568 16 3599 3583 0.0871906 0 - 0.36018 2024-04-22T06:58:32.815 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2569 16 3599 3583 0.0871566 0 - 0.36018 2024-04-22T06:58:33.815 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2570 16 3599 3583 0.0871227 0 - 0.36018 2024-04-22T06:58:34.815 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2571 16 3599 3583 0.0870888 0 - 0.36018 2024-04-22T06:58:35.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2572 16 3599 3583 0.087055 0 - 0.36018 2024-04-22T06:58:36.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2573 16 3599 3583 0.0870211 0 - 0.36018 2024-04-22T06:58:37.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2574 16 3599 3583 0.0869873 0 - 0.36018 2024-04-22T06:58:38.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2575 16 3599 3583 0.0869535 0 - 0.36018 2024-04-22T06:58:39.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2576 16 3599 3583 0.0869198 0 - 0.36018 2024-04-22T06:58:40.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2577 16 3599 3583 0.0868861 0 - 0.36018 2024-04-22T06:58:41.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2578 16 3599 3583 0.0868524 0 - 0.36018 2024-04-22T06:58:42.816 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2579 16 3599 3583 0.0868187 0 - 0.36018 2024-04-22T06:58:43.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:58:43.820023+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:58:43.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:58:43.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2580 16 3599 3583 0.086785 0 - 0.36018 2024-04-22T06:58:44.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2581 16 3599 3583 0.0867514 0 - 0.36018 2024-04-22T06:58:45.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2582 16 3599 3583 0.0867178 0 - 0.36018 2024-04-22T06:58:46.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2583 16 3599 3583 0.0866842 0 - 0.36018 2024-04-22T06:58:47.817 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2584 16 3599 3583 0.0866507 0 - 0.36018 2024-04-22T06:58:48.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2585 16 3599 3583 0.0866172 0 - 0.36018 2024-04-22T06:58:49.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2586 16 3599 3583 0.0865837 0 - 0.36018 2024-04-22T06:58:50.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2587 16 3599 3583 0.0865502 0 - 0.36018 2024-04-22T06:58:51.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2588 16 3599 3583 0.0865168 0 - 0.36018 2024-04-22T06:58:52.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2589 16 3599 3583 0.0864833 0 - 0.36018 2024-04-22T06:58:53.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2590 16 3599 3583 0.0864499 0 - 0.36018 2024-04-22T06:58:54.818 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2591 16 3599 3583 0.0864166 0 - 0.36018 2024-04-22T06:58:55.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2592 16 3599 3583 0.0863832 0 - 0.36018 2024-04-22T06:58:56.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2593 16 3599 3583 0.0863499 0 - 0.36018 2024-04-22T06:58:57.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2594 16 3599 3583 0.0863166 0 - 0.36018 2024-04-22T06:58:58.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2595 16 3599 3583 0.0862834 0 - 0.36018 2024-04-22T06:58:59.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2596 16 3599 3583 0.0862501 0 - 0.36018 2024-04-22T06:59:00.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2597 16 3599 3583 0.0862169 0 - 0.36018 2024-04-22T06:59:01.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2598 16 3599 3583 0.0861837 0 - 0.36018 2024-04-22T06:59:02.819 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2599 16 3599 3583 0.0861506 0 - 0.36018 2024-04-22T06:59:03.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:59:03.823029+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:59:03.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:59:03.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2600 16 3599 3583 0.0861174 0 - 0.36018 2024-04-22T06:59:04.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2601 16 3599 3583 0.0860843 0 - 0.36018 2024-04-22T06:59:05.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2602 16 3599 3583 0.0860512 0 - 0.36018 2024-04-22T06:59:06.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2603 16 3599 3583 0.0860182 0 - 0.36018 2024-04-22T06:59:07.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2604 16 3599 3583 0.0859852 0 - 0.36018 2024-04-22T06:59:08.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2605 16 3599 3583 0.0859521 0 - 0.36018 2024-04-22T06:59:09.820 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2606 16 3599 3583 0.0859192 0 - 0.36018 2024-04-22T06:59:10.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2607 16 3599 3583 0.0858862 0 - 0.36018 2024-04-22T06:59:11.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2608 16 3599 3583 0.0858533 0 - 0.36018 2024-04-22T06:59:12.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2609 16 3599 3583 0.0858204 0 - 0.36018 2024-04-22T06:59:13.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2610 16 3599 3583 0.0857875 0 - 0.36018 2024-04-22T06:59:14.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2611 16 3599 3583 0.0857546 0 - 0.36018 2024-04-22T06:59:15.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2612 16 3599 3583 0.0857218 0 - 0.36018 2024-04-22T06:59:16.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2613 16 3599 3583 0.085689 0 - 0.36018 2024-04-22T06:59:17.821 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2614 16 3599 3583 0.0856562 0 - 0.36018 2024-04-22T06:59:18.822 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2615 16 3599 3583 0.0856235 0 - 0.36018 2024-04-22T06:59:19.822 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2616 16 3599 3583 0.0855907 0 - 0.36018 2024-04-22T06:59:20.822 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2617 16 3599 3583 0.085558 0 - 0.36018 2024-04-22T06:59:21.822 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2618 16 3599 3583 0.0855253 0 - 0.36018 2024-04-22T06:59:22.822 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2619 16 3599 3583 0.0854927 0 - 0.36018 2024-04-22T06:59:23.822 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:59:23.825860+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:59:23.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:59:23.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2620 16 3599 3583 0.0854601 0 - 0.36018 2024-04-22T06:59:24.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2621 16 3599 3583 0.0854275 0 - 0.36018 2024-04-22T06:59:25.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2622 16 3599 3583 0.0853949 0 - 0.36018 2024-04-22T06:59:26.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2623 16 3599 3583 0.0853623 0 - 0.36018 2024-04-22T06:59:27.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2624 16 3599 3583 0.0853298 0 - 0.36018 2024-04-22T06:59:28.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2625 16 3599 3583 0.0852973 0 - 0.36018 2024-04-22T06:59:29.823 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2626 16 3599 3583 0.0852648 0 - 0.36018 2024-04-22T06:59:30.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2627 16 3599 3583 0.0852323 0 - 0.36018 2024-04-22T06:59:31.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2628 16 3599 3583 0.0851999 0 - 0.36018 2024-04-22T06:59:32.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2629 16 3599 3583 0.0851675 0 - 0.36018 2024-04-22T06:59:33.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2630 16 3599 3583 0.0851351 0 - 0.36018 2024-04-22T06:59:34.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2631 16 3599 3583 0.0851028 0 - 0.36018 2024-04-22T06:59:35.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2632 16 3599 3583 0.0850704 0 - 0.36018 2024-04-22T06:59:36.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2633 16 3599 3583 0.0850381 0 - 0.36018 2024-04-22T06:59:37.824 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2634 16 3599 3583 0.0850058 0 - 0.36018 2024-04-22T06:59:38.825 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2635 16 3599 3583 0.0849736 0 - 0.36018 2024-04-22T06:59:39.825 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2636 16 3599 3583 0.0849413 0 - 0.36018 2024-04-22T06:59:40.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2637 16 3599 3583 0.0849091 0 - 0.36018 2024-04-22T06:59:41.825 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2638 16 3599 3583 0.0848769 0 - 0.36018 2024-04-22T06:59:42.825 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2639 16 3599 3583 0.0848448 0 - 0.36018 2024-04-22T06:59:43.825 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T06:59:43.828802+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T06:59:43.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T06:59:43.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2640 16 3599 3583 0.0848126 0 - 0.36018 2024-04-22T06:59:44.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2641 16 3599 3583 0.0847805 0 - 0.36018 2024-04-22T06:59:45.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2642 16 3599 3583 0.0847484 0 - 0.36018 2024-04-22T06:59:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:59:46.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T06:59:46.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2643 16 3599 3583 0.0847164 0 - 0.36018 2024-04-22T06:59:47.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2644 16 3599 3583 0.0846843 0 - 0.36018 2024-04-22T06:59:48.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T06:59:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T06:59:48.695+0000 7f9f3e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:59:48.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2645 16 3599 3583 0.0846523 0 - 0.36018 2024-04-22T06:59:49.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2646 16 3599 3583 0.0846203 0 - 0.36018 2024-04-22T06:59:50.826 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2647 16 3599 3583 0.0845883 0 - 0.36018 2024-04-22T06:59:51.827 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2648 16 3599 3583 0.0845564 0 - 0.36018 2024-04-22T06:59:52.827 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2649 16 3599 3583 0.0845245 0 - 0.36018 2024-04-22T06:59:53.827 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2650 16 3599 3583 0.0844926 0 - 0.36018 2024-04-22T06:59:54.827 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2651 16 3599 3583 0.0844607 0 - 0.36018 2024-04-22T06:59:55.827 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2652 16 3599 3583 0.0844289 0 - 0.36018 2024-04-22T06:59:56.827 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2653 16 3599 3583 0.084397 0 - 0.36018 2024-04-22T06:59:57.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2654 16 3599 3583 0.0843652 0 - 0.36018 2024-04-22T06:59:58.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2655 16 3599 3583 0.0843335 0 - 0.36018 2024-04-22T06:59:59.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2656 16 3599 3583 0.0843017 0 - 0.36018 2024-04-22T07:00:00.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2657 16 3599 3583 0.08427 0 - 0.36018 2024-04-22T07:00:01.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2658 16 3599 3583 0.0842383 0 - 0.36018 2024-04-22T07:00:02.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2659 16 3599 3583 0.0842066 0 - 0.36018 2024-04-22T07:00:03.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:00:03.831710+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:00:03.828 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:00:03.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2660 16 3599 3583 0.0841749 0 - 0.36018 2024-04-22T07:00:04.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2661 16 3599 3583 0.0841433 0 - 0.36018 2024-04-22T07:00:05.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2662 16 3599 3583 0.0841117 0 - 0.36018 2024-04-22T07:00:06.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2663 16 3599 3583 0.0840801 0 - 0.36018 2024-04-22T07:00:07.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2664 16 3599 3583 0.0840486 0 - 0.36018 2024-04-22T07:00:08.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2665 16 3599 3583 0.084017 0 - 0.36018 2024-04-22T07:00:09.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2666 16 3599 3583 0.0839855 0 - 0.36018 2024-04-22T07:00:10.829 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2667 16 3599 3583 0.083954 0 - 0.36018 2024-04-22T07:00:11.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2668 16 3599 3583 0.0839225 0 - 0.36018 2024-04-22T07:00:12.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2669 16 3599 3583 0.0838911 0 - 0.36018 2024-04-22T07:00:13.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2670 16 3599 3583 0.0838597 0 - 0.36018 2024-04-22T07:00:14.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2671 16 3599 3583 0.0838283 0 - 0.36018 2024-04-22T07:00:15.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2672 16 3599 3583 0.0837969 0 - 0.36018 2024-04-22T07:00:16.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2673 16 3599 3583 0.0837656 0 - 0.36018 2024-04-22T07:00:17.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2674 16 3599 3583 0.0837342 0 - 0.36018 2024-04-22T07:00:18.830 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2675 16 3599 3583 0.0837029 0 - 0.36018 2024-04-22T07:00:19.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2676 16 3599 3583 0.0836717 0 - 0.36018 2024-04-22T07:00:20.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2677 16 3599 3583 0.0836404 0 - 0.36018 2024-04-22T07:00:21.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2678 16 3599 3583 0.0836092 0 - 0.36018 2024-04-22T07:00:22.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2679 16 3599 3583 0.083578 0 - 0.36018 2024-04-22T07:00:23.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:00:23.834641+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:00:23.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:00:23.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2680 16 3599 3583 0.0835468 0 - 0.36018 2024-04-22T07:00:24.831 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2681 16 3599 3583 0.0835156 0 - 0.36018 2024-04-22T07:00:25.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2682 16 3599 3583 0.0834845 0 - 0.36018 2024-04-22T07:00:26.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2683 16 3599 3583 0.0834533 0 - 0.36018 2024-04-22T07:00:27.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2684 16 3599 3583 0.0834223 0 - 0.36018 2024-04-22T07:00:28.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2685 16 3599 3583 0.0833912 0 - 0.36018 2024-04-22T07:00:29.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2686 16 3599 3583 0.0833601 0 - 0.36018 2024-04-22T07:00:30.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2687 16 3599 3583 0.0833291 0 - 0.36018 2024-04-22T07:00:31.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2688 16 3599 3583 0.0832981 0 - 0.36018 2024-04-22T07:00:32.832 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2689 16 3599 3583 0.0832671 0 - 0.36018 2024-04-22T07:00:33.833 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2690 16 3599 3583 0.0832362 0 - 0.36018 2024-04-22T07:00:34.833 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2691 16 3599 3583 0.0832053 0 - 0.36018 2024-04-22T07:00:35.833 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2692 16 3599 3583 0.0831743 0 - 0.36018 2024-04-22T07:00:36.833 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2693 16 3599 3583 0.0831435 0 - 0.36018 2024-04-22T07:00:37.833 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2694 16 3599 3583 0.0831126 0 - 0.36018 2024-04-22T07:00:38.833 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2695 16 3599 3583 0.0830818 0 - 0.36018 2024-04-22T07:00:39.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2696 16 3599 3583 0.0830509 0 - 0.36018 2024-04-22T07:00:40.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2697 16 3599 3583 0.0830201 0 - 0.36018 2024-04-22T07:00:41.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2698 16 3599 3583 0.0829894 0 - 0.36018 2024-04-22T07:00:42.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2699 16 3599 3583 0.0829586 0 - 0.36018 2024-04-22T07:00:43.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:00:43.837660+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:00:43.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:00:43.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2700 16 3599 3583 0.0829279 0 - 0.36018 2024-04-22T07:00:44.834 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2701 16 3599 3583 0.0828972 0 - 0.36018 2024-04-22T07:00:45.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2702 16 3599 3583 0.0828665 0 - 0.36018 2024-04-22T07:00:46.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2703 16 3599 3583 0.0828359 0 - 0.36018 2024-04-22T07:00:47.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2704 16 3599 3583 0.0828052 0 - 0.36018 2024-04-22T07:00:48.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2705 16 3599 3583 0.0827746 0 - 0.36018 2024-04-22T07:00:49.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2706 16 3599 3583 0.082744 0 - 0.36018 2024-04-22T07:00:50.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2707 16 3599 3583 0.0827135 0 - 0.36018 2024-04-22T07:00:51.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2708 16 3599 3583 0.0826829 0 - 0.36018 2024-04-22T07:00:52.835 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2709 16 3599 3583 0.0826524 0 - 0.36018 2024-04-22T07:00:53.836 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2710 16 3599 3583 0.0826219 0 - 0.36018 2024-04-22T07:00:54.836 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2711 16 3599 3583 0.0825914 0 - 0.36018 2024-04-22T07:00:55.836 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2712 16 3599 3583 0.082561 0 - 0.36018 2024-04-22T07:00:56.836 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2713 16 3599 3583 0.0825305 0 - 0.36018 2024-04-22T07:00:57.836 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2714 16 3599 3583 0.0825001 0 - 0.36018 2024-04-22T07:00:58.836 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2715 16 3599 3583 0.0824697 0 - 0.36018 2024-04-22T07:00:59.837 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2716 16 3599 3583 0.0824394 0 - 0.36018 2024-04-22T07:01:00.837 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2717 16 3599 3583 0.082409 0 - 0.36018 2024-04-22T07:01:01.837 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2718 16 3599 3583 0.0823787 0 - 0.36018 2024-04-22T07:01:02.837 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2719 16 3599 3583 0.0823484 0 - 0.36018 2024-04-22T07:01:03.837 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:01:03.840802+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:01:03.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:01:03.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2720 16 3599 3583 0.0823181 0 - 0.36018 2024-04-22T07:01:04.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2721 16 3599 3583 0.0822879 0 - 0.36018 2024-04-22T07:01:05.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2722 16 3599 3583 0.0822576 0 - 0.36018 2024-04-22T07:01:06.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2723 16 3599 3583 0.0822274 0 - 0.36018 2024-04-22T07:01:07.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2724 16 3599 3583 0.0821972 0 - 0.36018 2024-04-22T07:01:08.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2725 16 3599 3583 0.0821671 0 - 0.36018 2024-04-22T07:01:09.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2726 16 3599 3583 0.0821369 0 - 0.36018 2024-04-22T07:01:10.838 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2727 16 3599 3583 0.0821068 0 - 0.36018 2024-04-22T07:01:11.839 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2728 16 3599 3583 0.0820767 0 - 0.36018 2024-04-22T07:01:12.839 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2729 16 3599 3583 0.0820466 0 - 0.36018 2024-04-22T07:01:13.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2730 16 3599 3583 0.0820166 0 - 0.36018 2024-04-22T07:01:14.839 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2731 16 3599 3583 0.0819866 0 - 0.36018 2024-04-22T07:01:15.839 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2732 16 3599 3583 0.0819566 0 - 0.36018 2024-04-22T07:01:16.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2733 16 3599 3583 0.0819266 0 - 0.36018 2024-04-22T07:01:17.839 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2734 16 3599 3583 0.0818966 0 - 0.36018 2024-04-22T07:01:18.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2735 16 3599 3583 0.0818667 0 - 0.36018 2024-04-22T07:01:19.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2736 16 3599 3583 0.0818367 0 - 0.36018 2024-04-22T07:01:20.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2737 16 3599 3583 0.0818068 0 - 0.36018 2024-04-22T07:01:21.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2738 16 3599 3583 0.081777 0 - 0.36018 2024-04-22T07:01:22.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2739 16 3599 3583 0.0817471 0 - 0.36018 2024-04-22T07:01:23.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:01:23.843736+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:01:23.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:01:23.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2740 16 3599 3583 0.0817173 0 - 0.36018 2024-04-22T07:01:24.840 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2741 16 3599 3583 0.0816874 0 - 0.36018 2024-04-22T07:01:25.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2742 16 3599 3583 0.0816577 0 - 0.36018 2024-04-22T07:01:26.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2743 16 3599 3583 0.0816279 0 - 0.36018 2024-04-22T07:01:27.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2744 16 3599 3583 0.0815981 0 - 0.36018 2024-04-22T07:01:28.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2745 16 3599 3583 0.0815684 0 - 0.36018 2024-04-22T07:01:29.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2746 16 3599 3583 0.0815387 0 - 0.36018 2024-04-22T07:01:30.841 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2747 16 3599 3583 0.081509 0 - 0.36018 2024-04-22T07:01:31.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2748 16 3599 3583 0.0814794 0 - 0.36018 2024-04-22T07:01:32.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2749 16 3599 3583 0.0814497 0 - 0.36018 2024-04-22T07:01:33.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2750 16 3599 3583 0.0814201 0 - 0.36018 2024-04-22T07:01:34.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2751 16 3599 3583 0.0813905 0 - 0.36018 2024-04-22T07:01:35.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2752 16 3599 3583 0.0813609 0 - 0.36018 2024-04-22T07:01:36.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2753 16 3599 3583 0.0813314 0 - 0.36018 2024-04-22T07:01:37.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2754 16 3599 3583 0.0813019 0 - 0.36018 2024-04-22T07:01:38.842 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2755 16 3599 3583 0.0812723 0 - 0.36018 2024-04-22T07:01:39.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2756 16 3599 3583 0.0812429 0 - 0.36018 2024-04-22T07:01:40.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2757 16 3599 3583 0.0812134 0 - 0.36018 2024-04-22T07:01:41.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2758 16 3599 3583 0.0811839 0 - 0.36018 2024-04-22T07:01:42.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2759 16 3599 3583 0.0811545 0 - 0.36018 2024-04-22T07:01:43.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:01:43.846541+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:01:43.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:01:43.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2760 16 3599 3583 0.0811251 0 - 0.36018 2024-04-22T07:01:44.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2761 16 3599 3583 0.0810957 0 - 0.36018 2024-04-22T07:01:45.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2762 16 3599 3583 0.0810664 0 - 0.36018 2024-04-22T07:01:46.843 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2763 16 3599 3583 0.081037 0 - 0.36018 2024-04-22T07:01:47.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2764 16 3599 3583 0.0810077 0 - 0.36018 2024-04-22T07:01:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:01:48.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:01:48.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2765 16 3599 3583 0.0809784 0 - 0.36018 2024-04-22T07:01:49.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2766 16 3599 3583 0.0809491 0 - 0.36018 2024-04-22T07:01:50.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:01:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:01:50.737+0000 7fd3bf620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:01:50.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2767 16 3599 3583 0.0809199 0 - 0.36018 2024-04-22T07:01:51.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2768 16 3599 3583 0.0808906 0 - 0.36018 2024-04-22T07:01:52.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2769 16 3599 3583 0.0808614 0 - 0.36018 2024-04-22T07:01:53.844 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2770 16 3599 3583 0.0808322 0 - 0.36018 2024-04-22T07:01:54.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2771 16 3599 3583 0.0808031 0 - 0.36018 2024-04-22T07:01:55.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2772 16 3599 3583 0.0807739 0 - 0.36018 2024-04-22T07:01:56.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2773 16 3599 3583 0.0807448 0 - 0.36018 2024-04-22T07:01:57.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2774 16 3599 3583 0.0807157 0 - 0.36018 2024-04-22T07:01:58.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2775 16 3599 3583 0.0806866 0 - 0.36018 2024-04-22T07:01:59.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2776 16 3599 3583 0.0806575 0 - 0.36018 2024-04-22T07:02:00.845 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2777 16 3599 3583 0.0806285 0 - 0.36018 2024-04-22T07:02:01.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2778 16 3599 3583 0.0805995 0 - 0.36018 2024-04-22T07:02:02.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2779 16 3599 3583 0.0805705 0 - 0.36018 2024-04-22T07:02:03.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:02:03.849360+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:02:03.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:02:03.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2780 16 3599 3583 0.0805415 0 - 0.36018 2024-04-22T07:02:04.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2781 16 3599 3583 0.0805125 0 - 0.36018 2024-04-22T07:02:05.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2782 16 3599 3583 0.0804836 0 - 0.36018 2024-04-22T07:02:06.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2783 16 3599 3583 0.0804547 0 - 0.36018 2024-04-22T07:02:07.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2784 16 3599 3583 0.0804258 0 - 0.36018 2024-04-22T07:02:08.846 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2785 16 3599 3583 0.0803969 0 - 0.36018 2024-04-22T07:02:09.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2786 16 3599 3583 0.080368 0 - 0.36018 2024-04-22T07:02:10.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2787 16 3599 3583 0.0803392 0 - 0.36018 2024-04-22T07:02:11.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2788 16 3599 3583 0.0803104 0 - 0.36018 2024-04-22T07:02:12.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2789 16 3599 3583 0.0802816 0 - 0.36018 2024-04-22T07:02:13.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2790 16 3599 3583 0.0802528 0 - 0.36018 2024-04-22T07:02:14.847 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2791 16 3599 3583 0.080224 0 - 0.36018 2024-04-22T07:02:15.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2792 16 3599 3583 0.0801953 0 - 0.36018 2024-04-22T07:02:16.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2793 16 3599 3583 0.0801666 0 - 0.36018 2024-04-22T07:02:17.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2794 16 3599 3583 0.0801379 0 - 0.36018 2024-04-22T07:02:18.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2795 16 3599 3583 0.0801092 0 - 0.36018 2024-04-22T07:02:19.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2796 16 3599 3583 0.0800806 0 - 0.36018 2024-04-22T07:02:20.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2797 16 3599 3583 0.080052 0 - 0.36018 2024-04-22T07:02:21.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2798 16 3599 3583 0.0800233 0 - 0.36018 2024-04-22T07:02:22.848 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2799 16 3599 3583 0.0799948 0 - 0.36018 2024-04-22T07:02:23.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:02:23.852023+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:02:23.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:02:23.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2800 16 3599 3583 0.0799662 0 - 0.36018 2024-04-22T07:02:24.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2801 16 3599 3583 0.0799376 0 - 0.36018 2024-04-22T07:02:25.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2802 16 3599 3583 0.0799091 0 - 0.36018 2024-04-22T07:02:26.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2803 16 3599 3583 0.0798806 0 - 0.36018 2024-04-22T07:02:27.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2804 16 3599 3583 0.0798521 0 - 0.36018 2024-04-22T07:02:28.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2805 16 3599 3583 0.0798236 0 - 0.36018 2024-04-22T07:02:29.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2806 16 3599 3583 0.0797952 0 - 0.36018 2024-04-22T07:02:30.849 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2807 16 3599 3583 0.0797668 0 - 0.36018 2024-04-22T07:02:31.850 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2808 16 3599 3583 0.0797384 0 - 0.36018 2024-04-22T07:02:32.850 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2809 16 3599 3583 0.07971 0 - 0.36018 2024-04-22T07:02:33.850 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2810 16 3599 3583 0.0796816 0 - 0.36018 2024-04-22T07:02:34.850 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2811 16 3599 3583 0.0796533 0 - 0.36018 2024-04-22T07:02:35.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2812 16 3599 3583 0.0796249 0 - 0.36018 2024-04-22T07:02:36.850 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2813 16 3599 3583 0.0795966 0 - 0.36018 2024-04-22T07:02:37.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2814 16 3599 3583 0.0795683 0 - 0.36018 2024-04-22T07:02:38.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2815 16 3599 3583 0.0795401 0 - 0.36018 2024-04-22T07:02:39.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2816 16 3599 3583 0.0795118 0 - 0.36018 2024-04-22T07:02:40.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2817 16 3599 3583 0.0794836 0 - 0.36018 2024-04-22T07:02:41.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2818 16 3599 3583 0.0794554 0 - 0.36018 2024-04-22T07:02:42.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2819 16 3599 3583 0.0794272 0 - 0.36018 2024-04-22T07:02:43.851 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:02:43.854922+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:02:43.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:02:43.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2820 16 3599 3583 0.0793991 0 - 0.36018 2024-04-22T07:02:44.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2821 16 3599 3583 0.0793709 0 - 0.36018 2024-04-22T07:02:45.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2822 16 3599 3583 0.0793428 0 - 0.36018 2024-04-22T07:02:46.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2823 16 3599 3583 0.0793147 0 - 0.36018 2024-04-22T07:02:47.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2824 16 3599 3583 0.0792866 0 - 0.36018 2024-04-22T07:02:48.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2825 16 3599 3583 0.0792585 0 - 0.36018 2024-04-22T07:02:49.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2826 16 3599 3583 0.0792305 0 - 0.36018 2024-04-22T07:02:50.852 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2827 16 3599 3583 0.0792024 0 - 0.36018 2024-04-22T07:02:51.853 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2828 16 3599 3583 0.0791744 0 - 0.36018 2024-04-22T07:02:52.853 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2829 16 3599 3583 0.0791465 0 - 0.36018 2024-04-22T07:02:53.853 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2830 16 3599 3583 0.0791185 0 - 0.36018 2024-04-22T07:02:54.853 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2831 16 3599 3583 0.0790905 0 - 0.36018 2024-04-22T07:02:55.853 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2832 16 3599 3583 0.0790626 0 - 0.36018 2024-04-22T07:02:56.853 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2833 16 3599 3583 0.0790347 0 - 0.36018 2024-04-22T07:02:57.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2834 16 3599 3583 0.0790068 0 - 0.36018 2024-04-22T07:02:58.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2835 16 3599 3583 0.0789789 0 - 0.36018 2024-04-22T07:02:59.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2836 16 3599 3583 0.0789511 0 - 0.36018 2024-04-22T07:03:00.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2837 16 3599 3583 0.0789233 0 - 0.36018 2024-04-22T07:03:01.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2838 16 3599 3583 0.0788955 0 - 0.36018 2024-04-22T07:03:02.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2839 16 3599 3583 0.0788677 0 - 0.36018 2024-04-22T07:03:03.854 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:03:03.858018+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:03:03.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:03:03.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2840 16 3599 3583 0.0788399 0 - 0.36018 2024-04-22T07:03:04.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2841 16 3599 3583 0.0788121 0 - 0.36018 2024-04-22T07:03:05.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2842 16 3599 3583 0.0787844 0 - 0.36018 2024-04-22T07:03:06.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2843 16 3599 3583 0.0787567 0 - 0.36018 2024-04-22T07:03:07.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2844 16 3599 3583 0.078729 0 - 0.36018 2024-04-22T07:03:08.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2845 16 3599 3583 0.0787013 0 - 0.36018 2024-04-22T07:03:09.855 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2846 16 3599 3583 0.0786737 0 - 0.36018 2024-04-22T07:03:10.856 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2847 16 3599 3583 0.078646 0 - 0.36018 2024-04-22T07:03:11.856 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2848 16 3599 3583 0.0786184 0 - 0.36018 2024-04-22T07:03:12.856 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2849 16 3599 3583 0.0785908 0 - 0.36018 2024-04-22T07:03:13.856 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2850 16 3599 3583 0.0785633 0 - 0.36018 2024-04-22T07:03:14.856 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2851 16 3599 3583 0.0785357 0 - 0.36018 2024-04-22T07:03:15.856 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2852 16 3599 3583 0.0785082 0 - 0.36018 2024-04-22T07:03:16.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2853 16 3599 3583 0.0784807 0 - 0.36018 2024-04-22T07:03:17.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2854 16 3599 3583 0.0784532 0 - 0.36018 2024-04-22T07:03:18.859 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2855 16 3599 3583 0.0784257 0 - 0.36018 2024-04-22T07:03:19.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2856 16 3599 3583 0.0783982 0 - 0.36018 2024-04-22T07:03:20.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2857 16 3599 3583 0.0783708 0 - 0.36018 2024-04-22T07:03:21.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2858 16 3599 3583 0.0783434 0 - 0.36018 2024-04-22T07:03:22.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2859 16 3599 3583 0.078316 0 - 0.36018 2024-04-22T07:03:23.857 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:03:23.860997+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:03:23.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:03:23.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2860 16 3599 3583 0.0782886 0 - 0.36018 2024-04-22T07:03:24.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2861 16 3599 3583 0.0782612 0 - 0.36018 2024-04-22T07:03:25.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2862 16 3599 3583 0.0782339 0 - 0.36018 2024-04-22T07:03:26.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2863 16 3599 3583 0.0782065 0 - 0.36018 2024-04-22T07:03:27.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2864 16 3599 3583 0.0781792 0 - 0.36018 2024-04-22T07:03:28.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2865 16 3599 3583 0.0781519 0 - 0.36018 2024-04-22T07:03:29.859 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2866 16 3599 3583 0.0781247 0 - 0.36018 2024-04-22T07:03:30.858 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2867 16 3599 3583 0.0780974 0 - 0.36018 2024-04-22T07:03:31.859 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2868 16 3599 3583 0.0780702 0 - 0.36018 2024-04-22T07:03:32.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2869 16 3599 3583 0.078043 0 - 0.36018 2024-04-22T07:03:33.859 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2870 16 3599 3583 0.0780158 0 - 0.36018 2024-04-22T07:03:34.859 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2871 16 3599 3583 0.0779886 0 - 0.36018 2024-04-22T07:03:35.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2872 16 3599 3583 0.0779615 0 - 0.36018 2024-04-22T07:03:36.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2873 16 3599 3583 0.0779343 0 - 0.36018 2024-04-22T07:03:37.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2874 16 3599 3583 0.0779072 0 - 0.36018 2024-04-22T07:03:38.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2875 16 3599 3583 0.0778801 0 - 0.36018 2024-04-22T07:03:39.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2876 16 3599 3583 0.077853 0 - 0.36018 2024-04-22T07:03:40.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2877 16 3599 3583 0.077826 0 - 0.36018 2024-04-22T07:03:41.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2878 16 3599 3583 0.0777989 0 - 0.36018 2024-04-22T07:03:42.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2879 16 3599 3583 0.0777719 0 - 0.36018 2024-04-22T07:03:43.860 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:03:43.863913+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:03:43.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:03:43.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2880 16 3599 3583 0.0777449 0 - 0.36018 2024-04-22T07:03:44.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2881 16 3599 3583 0.0777179 0 - 0.36018 2024-04-22T07:03:45.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2882 16 3599 3583 0.0776909 0 - 0.36018 2024-04-22T07:03:46.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2883 16 3599 3583 0.077664 0 - 0.36018 2024-04-22T07:03:47.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2884 16 3599 3583 0.0776371 0 - 0.36018 2024-04-22T07:03:48.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2885 16 3599 3583 0.0776102 0 - 0.36018 2024-04-22T07:03:49.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2886 16 3599 3583 0.0775833 0 - 0.36018 2024-04-22T07:03:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:03:50.650 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:03:50.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2887 16 3599 3583 0.0775564 0 - 0.36018 2024-04-22T07:03:51.861 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2888 16 3599 3583 0.0775295 0 - 0.36018 2024-04-22T07:03:52.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:03:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:03:52.774+0000 7f92722ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:03:52.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2889 16 3599 3583 0.0775027 0 - 0.36018 2024-04-22T07:03:53.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2890 16 3599 3583 0.0774759 0 - 0.36018 2024-04-22T07:03:54.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2891 16 3599 3583 0.0774491 0 - 0.36018 2024-04-22T07:03:55.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2892 16 3599 3583 0.0774223 0 - 0.36018 2024-04-22T07:03:56.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2893 16 3599 3583 0.0773955 0 - 0.36018 2024-04-22T07:03:57.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2894 16 3599 3583 0.0773688 0 - 0.36018 2024-04-22T07:03:58.862 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2895 16 3599 3583 0.0773421 0 - 0.36018 2024-04-22T07:03:59.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2896 16 3599 3583 0.0773154 0 - 0.36018 2024-04-22T07:04:00.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2897 16 3599 3583 0.0772887 0 - 0.36018 2024-04-22T07:04:01.863 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2898 16 3599 3583 0.077262 0 - 0.36018 2024-04-22T07:04:02.863 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2899 16 3599 3583 0.0772354 0 - 0.36018 2024-04-22T07:04:03.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:04:03.866726+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:04:03.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:04:03.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2900 16 3599 3583 0.0772087 0 - 0.36018 2024-04-22T07:04:04.863 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2901 16 3599 3583 0.0771821 0 - 0.36018 2024-04-22T07:04:05.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2902 16 3599 3583 0.0771555 0 - 0.36018 2024-04-22T07:04:06.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2903 16 3599 3583 0.0771289 0 - 0.36018 2024-04-22T07:04:07.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2904 16 3599 3583 0.0771024 0 - 0.36018 2024-04-22T07:04:08.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2905 16 3599 3583 0.0770758 0 - 0.36018 2024-04-22T07:04:09.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2906 16 3599 3583 0.0770493 0 - 0.36018 2024-04-22T07:04:10.864 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2907 16 3599 3583 0.0770228 0 - 0.36018 2024-04-22T07:04:11.865 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2908 16 3599 3583 0.0769963 0 - 0.36018 2024-04-22T07:04:12.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2909 16 3599 3583 0.0769699 0 - 0.36018 2024-04-22T07:04:13.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2910 16 3599 3583 0.0769434 0 - 0.36018 2024-04-22T07:04:14.865 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2911 16 3599 3583 0.076917 0 - 0.36018 2024-04-22T07:04:15.865 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2912 16 3599 3583 0.0768906 0 - 0.36018 2024-04-22T07:04:16.865 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2913 16 3599 3583 0.0768642 0 - 0.36018 2024-04-22T07:04:17.865 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2914 16 3599 3583 0.0768378 0 - 0.36018 2024-04-22T07:04:18.865 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2915 16 3599 3583 0.0768114 0 - 0.36018 2024-04-22T07:04:19.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2916 16 3599 3583 0.0767851 0 - 0.36018 2024-04-22T07:04:20.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2917 16 3599 3583 0.0767588 0 - 0.36018 2024-04-22T07:04:21.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2918 16 3599 3583 0.0767325 0 - 0.36018 2024-04-22T07:04:22.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2919 16 3599 3583 0.0767062 0 - 0.36018 2024-04-22T07:04:23.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:04:23.869673+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:04:23.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:04:23.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2920 16 3599 3583 0.0766799 0 - 0.36018 2024-04-22T07:04:24.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2921 16 3599 3583 0.0766536 0 - 0.36018 2024-04-22T07:04:25.866 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2922 16 3599 3583 0.0766274 0 - 0.36018 2024-04-22T07:04:26.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2923 16 3599 3583 0.0766012 0 - 0.36018 2024-04-22T07:04:27.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2924 16 3599 3583 0.076575 0 - 0.36018 2024-04-22T07:04:28.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2925 16 3599 3583 0.0765488 0 - 0.36018 2024-04-22T07:04:29.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2926 16 3599 3583 0.0765227 0 - 0.36018 2024-04-22T07:04:30.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2927 16 3599 3583 0.0764965 0 - 0.36018 2024-04-22T07:04:31.867 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2928 16 3599 3583 0.0764704 0 - 0.36018 2024-04-22T07:04:32.868 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2929 16 3599 3583 0.0764443 0 - 0.36018 2024-04-22T07:04:33.868 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2930 16 3599 3583 0.0764182 0 - 0.36018 2024-04-22T07:04:34.868 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2931 16 3599 3583 0.0763921 0 - 0.36018 2024-04-22T07:04:35.868 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2932 16 3599 3583 0.0763661 0 - 0.36018 2024-04-22T07:04:36.868 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2933 16 3599 3583 0.07634 0 - 0.36018 2024-04-22T07:04:37.868 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2934 16 3599 3583 0.076314 0 - 0.36018 2024-04-22T07:04:38.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2935 16 3599 3583 0.076288 0 - 0.36018 2024-04-22T07:04:39.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2936 16 3599 3583 0.076262 0 - 0.36018 2024-04-22T07:04:40.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2937 16 3599 3583 0.0762361 0 - 0.36018 2024-04-22T07:04:41.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2938 16 3599 3583 0.0762101 0 - 0.36018 2024-04-22T07:04:42.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2939 16 3599 3583 0.0761842 0 - 0.36018 2024-04-22T07:04:43.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:04:43.872607+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:04:43.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:04:43.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2940 16 3599 3583 0.0761583 0 - 0.36018 2024-04-22T07:04:44.869 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2941 16 3599 3583 0.0761324 0 - 0.36018 2024-04-22T07:04:45.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2942 16 3599 3583 0.0761065 0 - 0.36018 2024-04-22T07:04:46.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2943 16 3599 3583 0.0760806 0 - 0.36018 2024-04-22T07:04:47.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2944 16 3599 3583 0.0760548 0 - 0.36018 2024-04-22T07:04:48.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2945 16 3599 3583 0.076029 0 - 0.36018 2024-04-22T07:04:49.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2946 16 3599 3583 0.0760032 0 - 0.36018 2024-04-22T07:04:50.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2947 16 3599 3583 0.0759774 0 - 0.36018 2024-04-22T07:04:51.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2948 16 3599 3583 0.0759516 0 - 0.36018 2024-04-22T07:04:52.870 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2949 16 3599 3583 0.0759258 0 - 0.36018 2024-04-22T07:04:53.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2950 16 3599 3583 0.0759001 0 - 0.36018 2024-04-22T07:04:54.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2951 16 3599 3583 0.0758744 0 - 0.36018 2024-04-22T07:04:55.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2952 16 3599 3583 0.0758487 0 - 0.36018 2024-04-22T07:04:56.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2953 16 3599 3583 0.075823 0 - 0.36018 2024-04-22T07:04:57.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2954 16 3599 3583 0.0757973 0 - 0.36018 2024-04-22T07:04:58.871 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2955 16 3599 3583 0.0757717 0 - 0.36018 2024-04-22T07:04:59.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2956 16 3599 3583 0.075746 0 - 0.36018 2024-04-22T07:05:00.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2957 16 3599 3583 0.0757204 0 - 0.36018 2024-04-22T07:05:01.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2958 16 3599 3583 0.0756948 0 - 0.36018 2024-04-22T07:05:02.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2959 16 3599 3583 0.0756692 0 - 0.36018 2024-04-22T07:05:03.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:05:03.875657+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:05:03.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:05:03.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2960 16 3599 3583 0.0756437 0 - 0.36018 2024-04-22T07:05:04.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2961 16 3599 3583 0.0756181 0 - 0.36018 2024-04-22T07:05:05.872 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2962 16 3599 3583 0.0755926 0 - 0.36018 2024-04-22T07:05:06.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2963 16 3599 3583 0.0755671 0 - 0.36018 2024-04-22T07:05:07.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2964 16 3599 3583 0.0755416 0 - 0.36018 2024-04-22T07:05:08.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2965 16 3599 3583 0.0755161 0 - 0.36018 2024-04-22T07:05:09.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2966 16 3599 3583 0.0754907 0 - 0.36018 2024-04-22T07:05:10.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2967 16 3599 3583 0.0754652 0 - 0.36018 2024-04-22T07:05:11.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2968 16 3599 3583 0.0754398 0 - 0.36018 2024-04-22T07:05:12.873 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2969 16 3599 3583 0.0754144 0 - 0.36018 2024-04-22T07:05:13.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2970 16 3599 3583 0.075389 0 - 0.36018 2024-04-22T07:05:14.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2971 16 3599 3583 0.0753636 0 - 0.36018 2024-04-22T07:05:15.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2972 16 3599 3583 0.0753382 0 - 0.36018 2024-04-22T07:05:16.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2973 16 3599 3583 0.0753129 0 - 0.36018 2024-04-22T07:05:17.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2974 16 3599 3583 0.0752876 0 - 0.36018 2024-04-22T07:05:18.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2975 16 3599 3583 0.0752623 0 - 0.36018 2024-04-22T07:05:19.874 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2976 16 3599 3583 0.075237 0 - 0.36018 2024-04-22T07:05:20.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2977 16 3599 3583 0.0752117 0 - 0.36018 2024-04-22T07:05:21.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2978 16 3599 3583 0.0751865 0 - 0.36018 2024-04-22T07:05:22.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2979 16 3599 3583 0.0751612 0 - 0.36018 2024-04-22T07:05:23.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:05:23.878501+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:05:23.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:05:23.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2980 16 3599 3583 0.075136 0 - 0.36018 2024-04-22T07:05:24.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2981 16 3599 3583 0.0751108 0 - 0.36018 2024-04-22T07:05:25.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2982 16 3599 3583 0.0750856 0 - 0.36018 2024-04-22T07:05:26.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2983 16 3599 3583 0.0750604 0 - 0.36018 2024-04-22T07:05:27.875 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2984 16 3599 3583 0.0750353 0 - 0.36018 2024-04-22T07:05:28.876 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2985 16 3599 3583 0.0750101 0 - 0.36018 2024-04-22T07:05:29.876 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2986 16 3599 3583 0.074985 0 - 0.36018 2024-04-22T07:05:30.876 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2987 16 3599 3583 0.0749599 0 - 0.36018 2024-04-22T07:05:31.876 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2988 16 3599 3583 0.0749348 0 - 0.36018 2024-04-22T07:05:32.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2989 16 3599 3583 0.0749098 0 - 0.36018 2024-04-22T07:05:33.876 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2990 16 3599 3583 0.0748847 0 - 0.36018 2024-04-22T07:05:34.876 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2991 16 3599 3583 0.0748597 0 - 0.36018 2024-04-22T07:05:35.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2992 16 3599 3583 0.0748347 0 - 0.36018 2024-04-22T07:05:36.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2993 16 3599 3583 0.0748097 0 - 0.36018 2024-04-22T07:05:37.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2994 16 3599 3583 0.0747847 0 - 0.36018 2024-04-22T07:05:38.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2995 16 3599 3583 0.0747597 0 - 0.36018 2024-04-22T07:05:39.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2996 16 3599 3583 0.0747347 0 - 0.36018 2024-04-22T07:05:40.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2997 16 3599 3583 0.0747098 0 - 0.36018 2024-04-22T07:05:41.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2998 16 3599 3583 0.0746849 0 - 0.36018 2024-04-22T07:05:42.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2999 16 3599 3583 0.07466 0 - 0.36018 2024-04-22T07:05:43.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:05:43.881389+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:05:43.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:05:43.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3000 16 3599 3583 0.0746351 0 - 0.36018 2024-04-22T07:05:44.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3001 16 3599 3583 0.0746102 0 - 0.36018 2024-04-22T07:05:45.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3002 16 3599 3583 0.0745854 0 - 0.36018 2024-04-22T07:05:46.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3003 16 3599 3583 0.0745605 0 - 0.36018 2024-04-22T07:05:47.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3004 16 3599 3583 0.0745357 0 - 0.36018 2024-04-22T07:05:48.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3005 16 3599 3583 0.0745109 0 - 0.36018 2024-04-22T07:05:49.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3006 16 3599 3583 0.0744861 0 - 0.36018 2024-04-22T07:05:50.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3007 16 3599 3583 0.0744613 0 - 0.36018 2024-04-22T07:05:51.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3008 16 3599 3583 0.0744366 0 - 0.36018 2024-04-22T07:05:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:05:52.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:05:52.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3009 16 3599 3583 0.0744119 0 - 0.36018 2024-04-22T07:05:53.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3010 16 3599 3583 0.0743871 0 - 0.36018 2024-04-22T07:05:54.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:05:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:05:54.812+0000 7fc73bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:05:54.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3011 16 3599 3583 0.0743624 0 - 0.36018 2024-04-22T07:05:55.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3012 16 3599 3583 0.0743377 0 - 0.36018 2024-04-22T07:05:56.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3013 16 3599 3583 0.0743131 0 - 0.36018 2024-04-22T07:05:57.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3014 16 3599 3583 0.0742884 0 - 0.36018 2024-04-22T07:05:58.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3015 16 3599 3583 0.0742638 0 - 0.36018 2024-04-22T07:05:59.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3016 16 3599 3583 0.0742391 0 - 0.36018 2024-04-22T07:06:00.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3017 16 3599 3583 0.0742145 0 - 0.36018 2024-04-22T07:06:01.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3018 16 3599 3583 0.07419 0 - 0.36018 2024-04-22T07:06:02.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3019 16 3599 3583 0.0741654 0 - 0.36018 2024-04-22T07:06:03.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:06:03.884285+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:06:03.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:06:03.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3020 16 3599 3583 0.0741408 0 - 0.36018 2024-04-22T07:06:04.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3021 16 3599 3583 0.0741163 0 - 0.36018 2024-04-22T07:06:05.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3022 16 3599 3583 0.0740918 0 - 0.36018 2024-04-22T07:06:06.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3023 16 3599 3583 0.0740672 0 - 0.36018 2024-04-22T07:06:07.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3024 16 3599 3583 0.0740428 0 - 0.36018 2024-04-22T07:06:08.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3025 16 3599 3583 0.0740183 0 - 0.36018 2024-04-22T07:06:09.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3026 16 3599 3583 0.0739938 0 - 0.36018 2024-04-22T07:06:10.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3027 16 3599 3583 0.0739694 0 - 0.36018 2024-04-22T07:06:11.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3028 16 3599 3583 0.0739449 0 - 0.36018 2024-04-22T07:06:12.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3029 16 3599 3583 0.0739205 0 - 0.36018 2024-04-22T07:06:13.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3030 16 3599 3583 0.0738961 0 - 0.36018 2024-04-22T07:06:14.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3031 16 3599 3583 0.0738718 0 - 0.36018 2024-04-22T07:06:15.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3032 16 3599 3583 0.0738474 0 - 0.36018 2024-04-22T07:06:16.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3033 16 3599 3583 0.073823 0 - 0.36018 2024-04-22T07:06:17.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3034 16 3599 3583 0.0737987 0 - 0.36018 2024-04-22T07:06:18.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3035 16 3599 3583 0.0737744 0 - 0.36018 2024-04-22T07:06:19.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3036 16 3599 3583 0.0737501 0 - 0.36018 2024-04-22T07:06:20.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3037 16 3599 3583 0.0737258 0 - 0.36018 2024-04-22T07:06:21.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3038 16 3599 3583 0.0737015 0 - 0.36018 2024-04-22T07:06:22.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3039 16 3599 3583 0.0736773 0 - 0.36018 2024-04-22T07:06:23.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:06:23.887133+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:06:23.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:06:23.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3040 16 3599 3583 0.0736531 0 - 0.36018 2024-04-22T07:06:24.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3041 16 3599 3583 0.0736288 0 - 0.36018 2024-04-22T07:06:25.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3042 16 3599 3583 0.0736046 0 - 0.36018 2024-04-22T07:06:26.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3043 16 3599 3583 0.0735804 0 - 0.36018 2024-04-22T07:06:27.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3044 16 3599 3583 0.0735563 0 - 0.36018 2024-04-22T07:06:28.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3045 16 3599 3583 0.0735321 0 - 0.36018 2024-04-22T07:06:29.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3046 16 3599 3583 0.073508 0 - 0.36018 2024-04-22T07:06:30.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3047 16 3599 3583 0.0734838 0 - 0.36018 2024-04-22T07:06:31.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3048 16 3599 3583 0.0734597 0 - 0.36018 2024-04-22T07:06:32.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3049 16 3599 3583 0.0734356 0 - 0.36018 2024-04-22T07:06:33.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3050 16 3599 3583 0.0734116 0 - 0.36018 2024-04-22T07:06:34.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3051 16 3599 3583 0.0733875 0 - 0.36018 2024-04-22T07:06:35.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3052 16 3599 3583 0.0733635 0 - 0.36018 2024-04-22T07:06:36.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3053 16 3599 3583 0.0733394 0 - 0.36018 2024-04-22T07:06:37.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3054 16 3599 3583 0.0733154 0 - 0.36018 2024-04-22T07:06:38.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3055 16 3599 3583 0.0732914 0 - 0.36018 2024-04-22T07:06:39.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3056 16 3599 3583 0.0732674 0 - 0.36018 2024-04-22T07:06:40.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3057 16 3599 3583 0.0732435 0 - 0.36018 2024-04-22T07:06:41.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3058 16 3599 3583 0.0732195 0 - 0.36018 2024-04-22T07:06:42.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3059 16 3599 3583 0.0731956 0 - 0.36018 2024-04-22T07:06:43.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:06:43.890052+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:06:43.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:06:43.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3060 16 3599 3583 0.0731717 0 - 0.36018 2024-04-22T07:06:44.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3061 16 3599 3583 0.0731478 0 - 0.36018 2024-04-22T07:06:45.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3062 16 3599 3583 0.0731239 0 - 0.36018 2024-04-22T07:06:46.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3063 16 3599 3583 0.0731 0 - 0.36018 2024-04-22T07:06:47.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3064 16 3599 3583 0.0730761 0 - 0.36018 2024-04-22T07:06:48.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3065 16 3599 3583 0.0730523 0 - 0.36018 2024-04-22T07:06:49.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3066 16 3599 3583 0.0730285 0 - 0.36018 2024-04-22T07:06:50.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3067 16 3599 3583 0.0730047 0 - 0.36018 2024-04-22T07:06:51.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3068 16 3599 3583 0.0729809 0 - 0.36018 2024-04-22T07:06:52.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3069 16 3599 3583 0.0729571 0 - 0.36018 2024-04-22T07:06:53.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3070 16 3599 3583 0.0729333 0 - 0.36018 2024-04-22T07:06:54.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3071 16 3599 3583 0.0729096 0 - 0.36018 2024-04-22T07:06:55.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3072 16 3599 3583 0.0728858 0 - 0.36018 2024-04-22T07:06:56.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3073 16 3599 3583 0.0728621 0 - 0.36018 2024-04-22T07:06:57.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3074 16 3599 3583 0.0728384 0 - 0.36018 2024-04-22T07:06:58.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3075 16 3599 3583 0.0728147 0 - 0.36018 2024-04-22T07:06:59.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3076 16 3599 3583 0.0727911 0 - 0.36018 2024-04-22T07:07:00.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3077 16 3599 3583 0.0727674 0 - 0.36018 2024-04-22T07:07:01.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3078 16 3599 3583 0.0727438 0 - 0.36018 2024-04-22T07:07:02.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3079 16 3599 3583 0.0727201 0 - 0.36018 2024-04-22T07:07:03.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:07:03.892827+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:07:03.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:07:03.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3080 16 3599 3583 0.0726965 0 - 0.36018 2024-04-22T07:07:04.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3081 16 3599 3583 0.0726729 0 - 0.36018 2024-04-22T07:07:05.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3082 16 3599 3583 0.0726493 0 - 0.36018 2024-04-22T07:07:06.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3083 16 3599 3583 0.0726258 0 - 0.36018 2024-04-22T07:07:07.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3084 16 3599 3583 0.0726022 0 - 0.36018 2024-04-22T07:07:08.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3085 16 3599 3583 0.0725787 0 - 0.36018 2024-04-22T07:07:09.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3086 16 3599 3583 0.0725552 0 - 0.36018 2024-04-22T07:07:10.890 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3087 16 3599 3583 0.0725317 0 - 0.36018 2024-04-22T07:07:11.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3088 16 3599 3583 0.0725082 0 - 0.36018 2024-04-22T07:07:12.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3089 16 3599 3583 0.0724847 0 - 0.36018 2024-04-22T07:07:13.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3090 16 3599 3583 0.0724613 0 - 0.36018 2024-04-22T07:07:14.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3091 16 3599 3583 0.0724378 0 - 0.36018 2024-04-22T07:07:15.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3092 16 3599 3583 0.0724144 0 - 0.36018 2024-04-22T07:07:16.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3093 16 3599 3583 0.072391 0 - 0.36018 2024-04-22T07:07:17.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3094 16 3599 3583 0.0723676 0 - 0.36018 2024-04-22T07:07:18.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3095 16 3599 3583 0.0723442 0 - 0.36018 2024-04-22T07:07:19.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3096 16 3599 3583 0.0723208 0 - 0.36018 2024-04-22T07:07:20.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3097 16 3599 3583 0.0722975 0 - 0.36018 2024-04-22T07:07:21.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3098 16 3599 3583 0.0722741 0 - 0.36018 2024-04-22T07:07:22.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3099 16 3599 3583 0.0722508 0 - 0.36018 2024-04-22T07:07:23.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:07:23.895707+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:07:23.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:07:23.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3100 16 3599 3583 0.0722275 0 - 0.36018 2024-04-22T07:07:24.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3101 16 3599 3583 0.0722042 0 - 0.36018 2024-04-22T07:07:25.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3102 16 3599 3583 0.0721809 0 - 0.36018 2024-04-22T07:07:26.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3103 16 3599 3583 0.0721577 0 - 0.36018 2024-04-22T07:07:27.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3104 16 3599 3583 0.0721344 0 - 0.36018 2024-04-22T07:07:28.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3105 16 3599 3583 0.0721112 0 - 0.36018 2024-04-22T07:07:29.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3106 16 3599 3583 0.072088 0 - 0.36018 2024-04-22T07:07:30.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3107 16 3599 3583 0.0720648 0 - 0.36018 2024-04-22T07:07:31.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3108 16 3599 3583 0.0720416 0 - 0.36018 2024-04-22T07:07:32.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3109 16 3599 3583 0.0720184 0 - 0.36018 2024-04-22T07:07:33.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3110 16 3599 3583 0.0719953 0 - 0.36018 2024-04-22T07:07:34.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3111 16 3599 3583 0.0719721 0 - 0.36018 2024-04-22T07:07:35.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3112 16 3599 3583 0.071949 0 - 0.36018 2024-04-22T07:07:36.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3113 16 3599 3583 0.0719259 0 - 0.36018 2024-04-22T07:07:37.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3114 16 3599 3583 0.0719028 0 - 0.36018 2024-04-22T07:07:38.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3115 16 3599 3583 0.0718797 0 - 0.36018 2024-04-22T07:07:39.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3116 16 3599 3583 0.0718566 0 - 0.36018 2024-04-22T07:07:40.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3117 16 3599 3583 0.0718336 0 - 0.36018 2024-04-22T07:07:41.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3118 16 3599 3583 0.0718105 0 - 0.36018 2024-04-22T07:07:42.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3119 16 3599 3583 0.0717875 0 - 0.36018 2024-04-22T07:07:43.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:07:43.898608+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:07:43.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:07:43.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3120 16 3599 3583 0.0717645 0 - 0.36018 2024-04-22T07:07:44.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3121 16 3599 3583 0.0717415 0 - 0.36018 2024-04-22T07:07:45.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3122 16 3599 3583 0.0717185 0 - 0.36018 2024-04-22T07:07:46.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3123 16 3599 3583 0.0716956 0 - 0.36018 2024-04-22T07:07:47.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3124 16 3599 3583 0.0716726 0 - 0.36018 2024-04-22T07:07:48.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3125 16 3599 3583 0.0716497 0 - 0.36018 2024-04-22T07:07:49.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3126 16 3599 3583 0.0716268 0 - 0.36018 2024-04-22T07:07:50.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3127 16 3599 3583 0.0716039 0 - 0.36018 2024-04-22T07:07:51.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3128 16 3599 3583 0.071581 0 - 0.36018 2024-04-22T07:07:52.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3129 16 3599 3583 0.0715581 0 - 0.36018 2024-04-22T07:07:53.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3130 16 3599 3583 0.0715352 0 - 0.36018 2024-04-22T07:07:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:07:54.724 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:07:54.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3131 16 3599 3583 0.0715124 0 - 0.36018 2024-04-22T07:07:55.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3132 16 3599 3583 0.0714896 0 - 0.36018 2024-04-22T07:07:56.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:07:56.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:07:56.849+0000 7f80310ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:07:56.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3133 16 3599 3583 0.0714667 0 - 0.36018 2024-04-22T07:07:57.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3134 16 3599 3583 0.0714439 0 - 0.36018 2024-04-22T07:07:58.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3135 16 3599 3583 0.0714211 0 - 0.36018 2024-04-22T07:07:59.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3136 16 3599 3583 0.0713984 0 - 0.36018 2024-04-22T07:08:00.897 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3137 16 3599 3583 0.0713756 0 - 0.36018 2024-04-22T07:08:01.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3138 16 3599 3583 0.0713529 0 - 0.36018 2024-04-22T07:08:02.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3139 16 3599 3583 0.0713301 0 - 0.36018 2024-04-22T07:08:03.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:08:03.901409+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:08:03.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:08:03.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3140 16 3599 3583 0.0713074 0 - 0.36018 2024-04-22T07:08:04.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3141 16 3599 3583 0.0712847 0 - 0.36018 2024-04-22T07:08:05.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3142 16 3599 3583 0.071262 0 - 0.36018 2024-04-22T07:08:06.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3143 16 3599 3583 0.0712394 0 - 0.36018 2024-04-22T07:08:07.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3144 16 3599 3583 0.0712167 0 - 0.36018 2024-04-22T07:08:08.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3145 16 3599 3583 0.0711941 0 - 0.36018 2024-04-22T07:08:09.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3146 16 3599 3583 0.0711714 0 - 0.36018 2024-04-22T07:08:10.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3147 16 3599 3583 0.0711488 0 - 0.36018 2024-04-22T07:08:11.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3148 16 3599 3583 0.0711262 0 - 0.36018 2024-04-22T07:08:12.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3149 16 3599 3583 0.0711036 0 - 0.36018 2024-04-22T07:08:13.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3150 16 3599 3583 0.071081 0 - 0.36018 2024-04-22T07:08:14.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3151 16 3599 3583 0.0710585 0 - 0.36018 2024-04-22T07:08:15.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3152 16 3599 3583 0.0710359 0 - 0.36018 2024-04-22T07:08:16.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3153 16 3599 3583 0.0710134 0 - 0.36018 2024-04-22T07:08:17.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3154 16 3599 3583 0.0709909 0 - 0.36018 2024-04-22T07:08:18.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3155 16 3599 3583 0.0709684 0 - 0.36018 2024-04-22T07:08:19.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3156 16 3599 3583 0.0709459 0 - 0.36018 2024-04-22T07:08:20.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3157 16 3599 3583 0.0709234 0 - 0.36018 2024-04-22T07:08:21.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3158 16 3599 3583 0.070901 0 - 0.36018 2024-04-22T07:08:22.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3159 16 3599 3583 0.0708785 0 - 0.36018 2024-04-22T07:08:23.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:08:23.904390+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:08:23.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:08:23.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3160 16 3599 3583 0.0708561 0 - 0.36018 2024-04-22T07:08:24.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3161 16 3599 3583 0.0708337 0 - 0.36018 2024-04-22T07:08:25.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3162 16 3599 3583 0.0708113 0 - 0.36018 2024-04-22T07:08:26.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3163 16 3599 3583 0.0707889 0 - 0.36018 2024-04-22T07:08:27.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3164 16 3599 3583 0.0707665 0 - 0.36018 2024-04-22T07:08:28.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3165 16 3599 3583 0.0707442 0 - 0.36018 2024-04-22T07:08:29.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3166 16 3599 3583 0.0707218 0 - 0.36018 2024-04-22T07:08:30.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3167 16 3599 3583 0.0706995 0 - 0.36018 2024-04-22T07:08:31.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3168 16 3599 3583 0.0706772 0 - 0.36018 2024-04-22T07:08:32.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3169 16 3599 3583 0.0706549 0 - 0.36018 2024-04-22T07:08:33.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3170 16 3599 3583 0.0706326 0 - 0.36018 2024-04-22T07:08:34.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3171 16 3599 3583 0.0706103 0 - 0.36018 2024-04-22T07:08:35.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3172 16 3599 3583 0.070588 0 - 0.36018 2024-04-22T07:08:36.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3173 16 3599 3583 0.0705658 0 - 0.36018 2024-04-22T07:08:37.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3174 16 3599 3583 0.0705436 0 - 0.36018 2024-04-22T07:08:38.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3175 16 3599 3583 0.0705213 0 - 0.36018 2024-04-22T07:08:39.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3176 16 3599 3583 0.0704991 0 - 0.36018 2024-04-22T07:08:40.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3177 16 3599 3583 0.070477 0 - 0.36018 2024-04-22T07:08:41.903 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3178 16 3599 3583 0.0704548 0 - 0.36018 2024-04-22T07:08:42.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3179 16 3599 3583 0.0704326 0 - 0.36018 2024-04-22T07:08:43.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:08:43.907248+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:08:43.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:08:43.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3180 16 3599 3583 0.0704105 0 - 0.36018 2024-04-22T07:08:44.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3181 16 3599 3583 0.0703883 0 - 0.36018 2024-04-22T07:08:45.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3182 16 3599 3583 0.0703662 0 - 0.36018 2024-04-22T07:08:46.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3183 16 3599 3583 0.0703441 0 - 0.36018 2024-04-22T07:08:47.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3184 16 3599 3583 0.070322 0 - 0.36018 2024-04-22T07:08:48.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3185 16 3599 3583 0.0702999 0 - 0.36018 2024-04-22T07:08:49.904 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3186 16 3599 3583 0.0702779 0 - 0.36018 2024-04-22T07:08:50.905 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3187 16 3599 3583 0.0702558 0 - 0.36018 2024-04-22T07:08:51.905 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3188 16 3599 3583 0.0702338 0 - 0.36018 2024-04-22T07:08:52.905 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3189 16 3599 3583 0.0702118 0 - 0.36018 2024-04-22T07:08:53.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3190 16 3599 3583 0.0701897 0 - 0.36018 2024-04-22T07:08:54.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3191 16 3599 3583 0.0701677 0 - 0.36018 2024-04-22T07:08:55.905 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3192 16 3599 3583 0.0701458 0 - 0.36018 2024-04-22T07:08:56.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3193 16 3599 3583 0.0701238 0 - 0.36018 2024-04-22T07:08:57.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3194 16 3599 3583 0.0701018 0 - 0.36018 2024-04-22T07:08:58.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3195 16 3599 3583 0.0700799 0 - 0.36018 2024-04-22T07:08:59.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3196 16 3599 3583 0.070058 0 - 0.36018 2024-04-22T07:09:00.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3197 16 3599 3583 0.0700361 0 - 0.36018 2024-04-22T07:09:01.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3198 16 3599 3583 0.0700142 0 - 0.36018 2024-04-22T07:09:02.906 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3199 16 3599 3583 0.0699923 0 - 0.36018 2024-04-22T07:09:03.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:09:03.910130+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:09:03.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:09:03.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3200 16 3599 3583 0.0699704 0 - 0.36018 2024-04-22T07:09:04.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3201 16 3599 3583 0.0699485 0 - 0.36018 2024-04-22T07:09:05.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3202 16 3599 3583 0.0699267 0 - 0.36018 2024-04-22T07:09:06.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3203 16 3599 3583 0.0699049 0 - 0.36018 2024-04-22T07:09:07.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3204 16 3599 3583 0.069883 0 - 0.36018 2024-04-22T07:09:08.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3205 16 3599 3583 0.0698612 0 - 0.36018 2024-04-22T07:09:09.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3206 16 3599 3583 0.0698395 0 - 0.36018 2024-04-22T07:09:10.907 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3207 16 3599 3583 0.0698177 0 - 0.36018 2024-04-22T07:09:11.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3208 16 3599 3583 0.0697959 0 - 0.36018 2024-04-22T07:09:12.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3209 16 3599 3583 0.0697742 0 - 0.36018 2024-04-22T07:09:13.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3210 16 3599 3583 0.0697524 0 - 0.36018 2024-04-22T07:09:14.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3211 16 3599 3583 0.0697307 0 - 0.36018 2024-04-22T07:09:15.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3212 16 3599 3583 0.069709 0 - 0.36018 2024-04-22T07:09:16.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3213 16 3599 3583 0.0696873 0 - 0.36018 2024-04-22T07:09:17.908 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3214 16 3599 3583 0.0696656 0 - 0.36018 2024-04-22T07:09:18.909 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3215 16 3599 3583 0.0696439 0 - 0.36018 2024-04-22T07:09:19.909 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3216 16 3599 3583 0.0696223 0 - 0.36018 2024-04-22T07:09:20.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3217 16 3599 3583 0.0696006 0 - 0.36018 2024-04-22T07:09:21.909 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3218 16 3599 3583 0.069579 0 - 0.36018 2024-04-22T07:09:22.909 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3219 16 3599 3583 0.0695574 0 - 0.36018 2024-04-22T07:09:23.909 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:09:23.912958+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:09:23.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:09:23.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3220 16 3599 3583 0.0695358 0 - 0.36018 2024-04-22T07:09:24.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3221 16 3599 3583 0.0695142 0 - 0.36018 2024-04-22T07:09:25.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3222 16 3599 3583 0.0694926 0 - 0.36018 2024-04-22T07:09:26.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3223 16 3599 3583 0.0694711 0 - 0.36018 2024-04-22T07:09:27.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3224 16 3599 3583 0.0694495 0 - 0.36018 2024-04-22T07:09:28.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3225 16 3599 3583 0.069428 0 - 0.36018 2024-04-22T07:09:29.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3226 16 3599 3583 0.0694065 0 - 0.36018 2024-04-22T07:09:30.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3227 16 3599 3583 0.069385 0 - 0.36018 2024-04-22T07:09:31.910 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3228 16 3599 3583 0.0693635 0 - 0.36018 2024-04-22T07:09:32.911 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3229 16 3599 3583 0.069342 0 - 0.36018 2024-04-22T07:09:33.911 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3230 16 3599 3583 0.0693205 0 - 0.36018 2024-04-22T07:09:34.911 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3231 16 3599 3583 0.0692991 0 - 0.36018 2024-04-22T07:09:35.911 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3232 16 3599 3583 0.0692776 0 - 0.36018 2024-04-22T07:09:36.911 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3233 16 3599 3583 0.0692562 0 - 0.36018 2024-04-22T07:09:37.911 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3234 16 3599 3583 0.0692348 0 - 0.36018 2024-04-22T07:09:38.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3235 16 3599 3583 0.0692134 0 - 0.36018 2024-04-22T07:09:39.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3236 16 3599 3583 0.069192 0 - 0.36018 2024-04-22T07:09:40.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3237 16 3599 3583 0.0691706 0 - 0.36018 2024-04-22T07:09:41.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3238 16 3599 3583 0.0691493 0 - 0.36018 2024-04-22T07:09:42.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3239 16 3599 3583 0.0691279 0 - 0.36018 2024-04-22T07:09:43.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:09:43.915833+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:09:43.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:09:43.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3240 16 3599 3583 0.0691066 0 - 0.36018 2024-04-22T07:09:44.912 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3241 16 3599 3583 0.0690852 0 - 0.36018 2024-04-22T07:09:45.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3242 16 3599 3583 0.0690639 0 - 0.36018 2024-04-22T07:09:46.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3243 16 3599 3583 0.0690426 0 - 0.36018 2024-04-22T07:09:47.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3244 16 3599 3583 0.0690214 0 - 0.36018 2024-04-22T07:09:48.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3245 16 3599 3583 0.0690001 0 - 0.36018 2024-04-22T07:09:49.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3246 16 3599 3583 0.0689788 0 - 0.36018 2024-04-22T07:09:50.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3247 16 3599 3583 0.0689576 0 - 0.36018 2024-04-22T07:09:51.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3248 16 3599 3583 0.0689364 0 - 0.36018 2024-04-22T07:09:52.913 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3249 16 3599 3583 0.0689151 0 - 0.36018 2024-04-22T07:09:53.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3250 16 3599 3583 0.0688939 0 - 0.36018 2024-04-22T07:09:54.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3251 16 3599 3583 0.0688727 0 - 0.36018 2024-04-22T07:09:55.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3252 16 3599 3583 0.0688516 0 - 0.36018 2024-04-22T07:09:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:09:56.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:09:56.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3253 16 3599 3583 0.0688304 0 - 0.36018 2024-04-22T07:09:57.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3254 16 3599 3583 0.0688092 0 - 0.36018 2024-04-22T07:09:58.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:09:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:09:58.886+0000 7fc279a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:09:58.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3255 16 3599 3583 0.0687881 0 - 0.36018 2024-04-22T07:09:59.914 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3256 16 3599 3583 0.068767 0 - 0.36018 2024-04-22T07:10:00.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3257 16 3599 3583 0.0687459 0 - 0.36018 2024-04-22T07:10:01.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3258 16 3599 3583 0.0687248 0 - 0.36018 2024-04-22T07:10:02.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3259 16 3599 3583 0.0687037 0 - 0.36018 2024-04-22T07:10:03.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:10:03.918707+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:10:03.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:10:03.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3260 16 3599 3583 0.0686826 0 - 0.36018 2024-04-22T07:10:04.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3261 16 3599 3583 0.0686615 0 - 0.36018 2024-04-22T07:10:05.915 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3262 16 3599 3583 0.0686405 0 - 0.36018 2024-04-22T07:10:06.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3263 16 3599 3583 0.0686195 0 - 0.36018 2024-04-22T07:10:07.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3264 16 3599 3583 0.0685984 0 - 0.36018 2024-04-22T07:10:08.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3265 16 3599 3583 0.0685774 0 - 0.36018 2024-04-22T07:10:09.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3266 16 3599 3583 0.0685564 0 - 0.36018 2024-04-22T07:10:10.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3267 16 3599 3583 0.0685354 0 - 0.36018 2024-04-22T07:10:11.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3268 16 3599 3583 0.0685145 0 - 0.36018 2024-04-22T07:10:12.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3269 16 3599 3583 0.0684935 0 - 0.36018 2024-04-22T07:10:13.916 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3270 16 3599 3583 0.0684726 0 - 0.36018 2024-04-22T07:10:14.917 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3271 16 3599 3583 0.0684516 0 - 0.36018 2024-04-22T07:10:15.917 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3272 16 3599 3583 0.0684307 0 - 0.36018 2024-04-22T07:10:16.917 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3273 16 3599 3583 0.0684098 0 - 0.36018 2024-04-22T07:10:17.917 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3274 16 3599 3583 0.0683889 0 - 0.36018 2024-04-22T07:10:18.917 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3275 16 3599 3583 0.068368 0 - 0.36018 2024-04-22T07:10:19.917 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3276 16 3599 3583 0.0683472 0 - 0.36018 2024-04-22T07:10:20.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3277 16 3599 3583 0.0683263 0 - 0.36018 2024-04-22T07:10:21.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3278 16 3599 3583 0.0683055 0 - 0.36018 2024-04-22T07:10:22.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3279 16 3599 3583 0.0682846 0 - 0.36018 2024-04-22T07:10:23.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:10:23.921630+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:10:23.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:10:23.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3280 16 3599 3583 0.0682638 0 - 0.36018 2024-04-22T07:10:24.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3281 16 3599 3583 0.068243 0 - 0.36018 2024-04-22T07:10:25.918 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3282 16 3599 3583 0.0682222 0 - 0.36018 2024-04-22T07:10:26.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3283 16 3599 3583 0.0682014 0 - 0.36018 2024-04-22T07:10:27.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3284 16 3599 3583 0.0681807 0 - 0.36018 2024-04-22T07:10:28.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3285 16 3599 3583 0.0681599 0 - 0.36018 2024-04-22T07:10:29.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3286 16 3599 3583 0.0681392 0 - 0.36018 2024-04-22T07:10:30.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3287 16 3599 3583 0.0681184 0 - 0.36018 2024-04-22T07:10:31.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3288 16 3599 3583 0.0680977 0 - 0.36018 2024-04-22T07:10:32.919 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3289 16 3599 3583 0.068077 0 - 0.36018 2024-04-22T07:10:33.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3290 16 3599 3583 0.0680563 0 - 0.36018 2024-04-22T07:10:34.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3291 16 3599 3583 0.0680356 0 - 0.36018 2024-04-22T07:10:35.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3292 16 3599 3583 0.068015 0 - 0.36018 2024-04-22T07:10:36.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3293 16 3599 3583 0.0679943 0 - 0.36018 2024-04-22T07:10:37.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3294 16 3599 3583 0.0679737 0 - 0.36018 2024-04-22T07:10:38.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3295 16 3599 3583 0.067953 0 - 0.36018 2024-04-22T07:10:39.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3296 16 3599 3583 0.0679324 0 - 0.36018 2024-04-22T07:10:40.920 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3297 16 3599 3583 0.0679118 0 - 0.36018 2024-04-22T07:10:41.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3298 16 3599 3583 0.0678912 0 - 0.36018 2024-04-22T07:10:42.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3299 16 3599 3583 0.0678707 0 - 0.36018 2024-04-22T07:10:43.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:10:43.924463+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:10:43.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:10:43.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3300 16 3599 3583 0.0678501 0 - 0.36018 2024-04-22T07:10:44.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3301 16 3599 3583 0.0678295 0 - 0.36018 2024-04-22T07:10:45.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3302 16 3599 3583 0.067809 0 - 0.36018 2024-04-22T07:10:46.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3303 16 3599 3583 0.0677885 0 - 0.36018 2024-04-22T07:10:47.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3304 16 3599 3583 0.0677679 0 - 0.36018 2024-04-22T07:10:48.921 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3305 16 3599 3583 0.0677474 0 - 0.36018 2024-04-22T07:10:49.922 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3306 16 3599 3583 0.0677269 0 - 0.36018 2024-04-22T07:10:50.922 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3307 16 3599 3583 0.0677065 0 - 0.36018 2024-04-22T07:10:51.922 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3308 16 3599 3583 0.067686 0 - 0.36018 2024-04-22T07:10:52.922 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3309 16 3599 3583 0.0676655 0 - 0.36018 2024-04-22T07:10:53.922 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3310 16 3599 3583 0.0676451 0 - 0.36018 2024-04-22T07:10:54.922 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3311 16 3599 3583 0.0676247 0 - 0.36018 2024-04-22T07:10:55.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3312 16 3599 3583 0.0676043 0 - 0.36018 2024-04-22T07:10:56.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3313 16 3599 3583 0.0675838 0 - 0.36018 2024-04-22T07:10:57.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3314 16 3599 3583 0.0675635 0 - 0.36018 2024-04-22T07:10:58.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3315 16 3599 3583 0.0675431 0 - 0.36018 2024-04-22T07:10:59.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3316 16 3599 3583 0.0675227 0 - 0.36018 2024-04-22T07:11:00.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3317 16 3599 3583 0.0675023 0 - 0.36018 2024-04-22T07:11:01.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3318 16 3599 3583 0.067482 0 - 0.36018 2024-04-22T07:11:02.923 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3319 16 3599 3583 0.0674617 0 - 0.36018 2024-04-22T07:11:03.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:11:03.927220+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:11:03.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:11:03.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3320 16 3599 3583 0.0674414 0 - 0.36018 2024-04-22T07:11:04.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3321 16 3599 3583 0.067421 0 - 0.36018 2024-04-22T07:11:05.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3322 16 3599 3583 0.0674008 0 - 0.36018 2024-04-22T07:11:06.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3323 16 3599 3583 0.0673805 0 - 0.36018 2024-04-22T07:11:07.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3324 16 3599 3583 0.0673602 0 - 0.36018 2024-04-22T07:11:08.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3325 16 3599 3583 0.0673399 0 - 0.36018 2024-04-22T07:11:09.924 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3326 16 3599 3583 0.0673197 0 - 0.36018 2024-04-22T07:11:10.925 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3327 16 3599 3583 0.0672995 0 - 0.36018 2024-04-22T07:11:11.925 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3328 16 3599 3583 0.0672792 0 - 0.36018 2024-04-22T07:11:12.925 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3329 16 3599 3583 0.067259 0 - 0.36018 2024-04-22T07:11:13.925 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3330 16 3599 3583 0.0672388 0 - 0.36018 2024-04-22T07:11:14.925 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3331 16 3599 3583 0.0672186 0 - 0.36018 2024-04-22T07:11:15.925 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3332 16 3599 3583 0.0671985 0 - 0.36018 2024-04-22T07:11:16.926 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3333 16 3599 3583 0.0671783 0 - 0.36018 2024-04-22T07:11:17.926 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3334 16 3599 3583 0.0671582 0 - 0.36018 2024-04-22T07:11:18.926 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3335 16 3599 3583 0.067138 0 - 0.36018 2024-04-22T07:11:19.926 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3336 16 3599 3583 0.0671179 0 - 0.36018 2024-04-22T07:11:20.926 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3337 16 3599 3583 0.0670978 0 - 0.36018 2024-04-22T07:11:21.926 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3338 16 3599 3583 0.0670777 0 - 0.36018 2024-04-22T07:11:22.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3339 16 3599 3583 0.0670576 0 - 0.36018 2024-04-22T07:11:23.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:11:23.930262+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:11:23.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:11:23.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3340 16 3599 3583 0.0670375 0 - 0.36018 2024-04-22T07:11:24.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3341 16 3599 3583 0.0670174 0 - 0.36018 2024-04-22T07:11:25.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3342 16 3599 3583 0.0669974 0 - 0.36018 2024-04-22T07:11:26.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3343 16 3599 3583 0.0669774 0 - 0.36018 2024-04-22T07:11:27.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3344 16 3599 3583 0.0669573 0 - 0.36018 2024-04-22T07:11:28.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3345 16 3599 3583 0.0669373 0 - 0.36018 2024-04-22T07:11:29.927 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3346 16 3599 3583 0.0669173 0 - 0.36018 2024-04-22T07:11:30.928 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3347 16 3599 3583 0.0668973 0 - 0.36018 2024-04-22T07:11:31.928 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3348 16 3599 3583 0.0668773 0 - 0.36018 2024-04-22T07:11:32.928 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3349 16 3599 3583 0.0668574 0 - 0.36018 2024-04-22T07:11:33.928 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3350 16 3599 3583 0.0668374 0 - 0.36018 2024-04-22T07:11:34.928 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3351 16 3599 3583 0.0668175 0 - 0.36018 2024-04-22T07:11:35.928 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3352 16 3599 3583 0.0667975 0 - 0.36018 2024-04-22T07:11:36.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3353 16 3599 3583 0.0667776 0 - 0.36018 2024-04-22T07:11:37.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3354 16 3599 3583 0.0667577 0 - 0.36018 2024-04-22T07:11:38.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3355 16 3599 3583 0.0667378 0 - 0.36018 2024-04-22T07:11:39.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3356 16 3599 3583 0.0667179 0 - 0.36018 2024-04-22T07:11:40.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3357 16 3599 3583 0.066698 0 - 0.36018 2024-04-22T07:11:41.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3358 16 3599 3583 0.0666782 0 - 0.36018 2024-04-22T07:11:42.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3359 16 3599 3583 0.0666583 0 - 0.36018 2024-04-22T07:11:43.929 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:11:43.933102+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:11:43.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:11:43.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3360 16 3599 3583 0.0666385 0 - 0.36018 2024-04-22T07:11:44.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3361 16 3599 3583 0.0666187 0 - 0.36018 2024-04-22T07:11:45.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3362 16 3599 3583 0.0665988 0 - 0.36018 2024-04-22T07:11:46.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3363 16 3599 3583 0.066579 0 - 0.36018 2024-04-22T07:11:47.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3364 16 3599 3583 0.0665592 0 - 0.36018 2024-04-22T07:11:48.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3365 16 3599 3583 0.0665395 0 - 0.36018 2024-04-22T07:11:49.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3366 16 3599 3583 0.0665197 0 - 0.36018 2024-04-22T07:11:50.930 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3367 16 3599 3583 0.0664999 0 - 0.36018 2024-04-22T07:11:51.931 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3368 16 3599 3583 0.0664802 0 - 0.36018 2024-04-22T07:11:52.931 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3369 16 3599 3583 0.0664605 0 - 0.36018 2024-04-22T07:11:53.931 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3370 16 3599 3583 0.0664407 0 - 0.36018 2024-04-22T07:11:54.931 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3371 16 3599 3583 0.066421 0 - 0.36018 2024-04-22T07:11:55.931 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3372 16 3599 3583 0.0664013 0 - 0.36018 2024-04-22T07:11:56.931 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3373 16 3599 3583 0.0663816 0 - 0.36018 2024-04-22T07:11:57.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3374 16 3599 3583 0.066362 0 - 0.36018 2024-04-22T07:11:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:11:58.798 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:11:58.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3375 16 3599 3583 0.0663423 0 - 0.36018 2024-04-22T07:11:59.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3376 16 3599 3583 0.0663227 0 - 0.36018 2024-04-22T07:12:00.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:12:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:12:00.923+0000 7f6f1bced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:12:00.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3377 16 3599 3583 0.066303 0 - 0.36018 2024-04-22T07:12:01.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3378 16 3599 3583 0.0662834 0 - 0.36018 2024-04-22T07:12:02.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3379 16 3599 3583 0.0662638 0 - 0.36018 2024-04-22T07:12:03.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:12:03.935956+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:12:03.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:12:03.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3380 16 3599 3583 0.0662442 0 - 0.36018 2024-04-22T07:12:04.932 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3381 16 3599 3583 0.0662246 0 - 0.36018 2024-04-22T07:12:05.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3382 16 3599 3583 0.066205 0 - 0.36018 2024-04-22T07:12:06.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3383 16 3599 3583 0.0661854 0 - 0.36018 2024-04-22T07:12:07.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3384 16 3599 3583 0.0661659 0 - 0.36018 2024-04-22T07:12:08.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3385 16 3599 3583 0.0661463 0 - 0.36018 2024-04-22T07:12:09.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3386 16 3599 3583 0.0661268 0 - 0.36018 2024-04-22T07:12:10.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3387 16 3599 3583 0.0661073 0 - 0.36018 2024-04-22T07:12:11.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3388 16 3599 3583 0.0660877 0 - 0.36018 2024-04-22T07:12:12.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3389 16 3599 3583 0.0660682 0 - 0.36018 2024-04-22T07:12:13.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3390 16 3599 3583 0.0660488 0 - 0.36018 2024-04-22T07:12:14.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3391 16 3599 3583 0.0660293 0 - 0.36018 2024-04-22T07:12:15.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3392 16 3599 3583 0.0660098 0 - 0.36018 2024-04-22T07:12:16.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3393 16 3599 3583 0.0659904 0 - 0.36018 2024-04-22T07:12:17.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3394 16 3599 3583 0.0659709 0 - 0.36018 2024-04-22T07:12:18.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3395 16 3599 3583 0.0659515 0 - 0.36018 2024-04-22T07:12:19.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3396 16 3599 3583 0.0659321 0 - 0.36018 2024-04-22T07:12:20.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3397 16 3599 3583 0.0659127 0 - 0.36018 2024-04-22T07:12:21.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3398 16 3599 3583 0.0658933 0 - 0.36018 2024-04-22T07:12:22.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3399 16 3599 3583 0.0658739 0 - 0.36018 2024-04-22T07:12:23.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:12:23.938781+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:12:23.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:12:23.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3400 16 3599 3583 0.0658545 0 - 0.36018 2024-04-22T07:12:24.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3401 16 3599 3583 0.0658351 0 - 0.36018 2024-04-22T07:12:25.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3402 16 3599 3583 0.0658158 0 - 0.36018 2024-04-22T07:12:26.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3403 16 3599 3583 0.0657964 0 - 0.36018 2024-04-22T07:12:27.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3404 16 3599 3583 0.0657771 0 - 0.36018 2024-04-22T07:12:28.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3405 16 3599 3583 0.0657578 0 - 0.36018 2024-04-22T07:12:29.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3406 16 3599 3583 0.0657385 0 - 0.36018 2024-04-22T07:12:30.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3407 16 3599 3583 0.0657192 0 - 0.36018 2024-04-22T07:12:31.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3408 16 3599 3583 0.0656999 0 - 0.36018 2024-04-22T07:12:32.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3409 16 3599 3583 0.0656806 0 - 0.36018 2024-04-22T07:12:33.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3410 16 3599 3583 0.0656614 0 - 0.36018 2024-04-22T07:12:34.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3411 16 3599 3583 0.0656421 0 - 0.36018 2024-04-22T07:12:35.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3412 16 3599 3583 0.0656229 0 - 0.36018 2024-04-22T07:12:36.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3413 16 3599 3583 0.0656037 0 - 0.36018 2024-04-22T07:12:37.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3414 16 3599 3583 0.0655844 0 - 0.36018 2024-04-22T07:12:38.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3415 16 3599 3583 0.0655652 0 - 0.36018 2024-04-22T07:12:39.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3416 16 3599 3583 0.065546 0 - 0.36018 2024-04-22T07:12:40.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3417 16 3599 3583 0.0655269 0 - 0.36018 2024-04-22T07:12:41.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3418 16 3599 3583 0.0655077 0 - 0.36018 2024-04-22T07:12:42.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3419 16 3599 3583 0.0654885 0 - 0.36018 2024-04-22T07:12:43.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:12:43.941282+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:12:43.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:12:43.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3420 16 3599 3583 0.0654694 0 - 0.36018 2024-04-22T07:12:44.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3421 16 3599 3583 0.0654503 0 - 0.36018 2024-04-22T07:12:45.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3422 16 3599 3583 0.0654311 0 - 0.36018 2024-04-22T07:12:46.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3423 16 3599 3583 0.065412 0 - 0.36018 2024-04-22T07:12:47.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3424 16 3599 3583 0.0653929 0 - 0.36018 2024-04-22T07:12:48.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3425 16 3599 3583 0.0653738 0 - 0.36018 2024-04-22T07:12:49.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3426 16 3599 3583 0.0653547 0 - 0.36018 2024-04-22T07:12:50.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3427 16 3599 3583 0.0653357 0 - 0.36018 2024-04-22T07:12:51.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3428 16 3599 3583 0.0653166 0 - 0.36018 2024-04-22T07:12:52.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3429 16 3599 3583 0.0652976 0 - 0.36018 2024-04-22T07:12:53.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3430 16 3599 3583 0.0652785 0 - 0.36018 2024-04-22T07:12:54.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3431 16 3599 3583 0.0652595 0 - 0.36018 2024-04-22T07:12:55.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3432 16 3599 3583 0.0652405 0 - 0.36018 2024-04-22T07:12:56.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3433 16 3599 3583 0.0652215 0 - 0.36018 2024-04-22T07:12:57.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3434 16 3599 3583 0.0652025 0 - 0.36018 2024-04-22T07:12:58.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3435 16 3599 3583 0.0651835 0 - 0.36018 2024-04-22T07:12:59.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3436 16 3599 3583 0.0651645 0 - 0.36018 2024-04-22T07:13:00.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3437 16 3599 3583 0.0651456 0 - 0.36018 2024-04-22T07:13:01.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3438 16 3599 3583 0.0651266 0 - 0.36018 2024-04-22T07:13:02.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3439 16 3599 3583 0.0651077 0 - 0.36018 2024-04-22T07:13:03.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:13:03.944188+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:13:03.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:13:03.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3440 16 3599 3583 0.0650888 0 - 0.36018 2024-04-22T07:13:04.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3441 16 3599 3583 0.0650698 0 - 0.36018 2024-04-22T07:13:05.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3442 16 3599 3583 0.0650509 0 - 0.36018 2024-04-22T07:13:06.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3443 16 3599 3583 0.065032 0 - 0.36018 2024-04-22T07:13:07.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3444 16 3599 3583 0.0650132 0 - 0.36018 2024-04-22T07:13:08.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3445 16 3599 3583 0.0649943 0 - 0.36018 2024-04-22T07:13:09.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3446 16 3599 3583 0.0649754 0 - 0.36018 2024-04-22T07:13:10.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3447 16 3599 3583 0.0649566 0 - 0.36018 2024-04-22T07:13:11.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3448 16 3599 3583 0.0649377 0 - 0.36018 2024-04-22T07:13:12.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3449 16 3599 3583 0.0649189 0 - 0.36018 2024-04-22T07:13:13.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3450 16 3599 3583 0.0649001 0 - 0.36018 2024-04-22T07:13:14.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3451 16 3599 3583 0.0648813 0 - 0.36018 2024-04-22T07:13:15.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3452 16 3599 3583 0.0648625 0 - 0.36018 2024-04-22T07:13:16.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3453 16 3599 3583 0.0648437 0 - 0.36018 2024-04-22T07:13:17.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3454 16 3599 3583 0.0648249 0 - 0.36018 2024-04-22T07:13:18.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3455 16 3599 3583 0.0648062 0 - 0.36018 2024-04-22T07:13:19.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3456 16 3599 3583 0.0647874 0 - 0.36018 2024-04-22T07:13:20.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3457 16 3599 3583 0.0647687 0 - 0.36018 2024-04-22T07:13:21.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3458 16 3599 3583 0.0647499 0 - 0.36018 2024-04-22T07:13:22.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3459 16 3599 3583 0.0647312 0 - 0.36018 2024-04-22T07:13:23.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:13:23.946973+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:13:23.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:13:23.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3460 16 3599 3583 0.0647125 0 - 0.36018 2024-04-22T07:13:24.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3461 16 3599 3583 0.0646938 0 - 0.36018 2024-04-22T07:13:25.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3462 16 3599 3583 0.0646751 0 - 0.36018 2024-04-22T07:13:26.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3463 16 3599 3583 0.0646565 0 - 0.36018 2024-04-22T07:13:27.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3464 16 3599 3583 0.0646378 0 - 0.36018 2024-04-22T07:13:28.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3465 16 3599 3583 0.0646191 0 - 0.36018 2024-04-22T07:13:29.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3466 16 3599 3583 0.0646005 0 - 0.36018 2024-04-22T07:13:30.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3467 16 3599 3583 0.0645819 0 - 0.36018 2024-04-22T07:13:31.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3468 16 3599 3583 0.0645632 0 - 0.36018 2024-04-22T07:13:32.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3469 16 3599 3583 0.0645446 0 - 0.36018 2024-04-22T07:13:33.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3470 16 3599 3583 0.064526 0 - 0.36018 2024-04-22T07:13:34.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3471 16 3599 3583 0.0645074 0 - 0.36018 2024-04-22T07:13:35.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3472 16 3599 3583 0.0644889 0 - 0.36018 2024-04-22T07:13:36.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3473 16 3599 3583 0.0644703 0 - 0.36018 2024-04-22T07:13:37.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3474 16 3599 3583 0.0644517 0 - 0.36018 2024-04-22T07:13:38.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3475 16 3599 3583 0.0644332 0 - 0.36018 2024-04-22T07:13:39.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3476 16 3599 3583 0.0644146 0 - 0.36018 2024-04-22T07:13:40.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3477 16 3599 3583 0.0643961 0 - 0.36018 2024-04-22T07:13:41.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3478 16 3599 3583 0.0643776 0 - 0.36018 2024-04-22T07:13:42.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3479 16 3599 3583 0.0643591 0 - 0.36018 2024-04-22T07:13:43.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:13:43.949807+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:13:43.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:13:43.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3480 16 3599 3583 0.0643406 0 - 0.36018 2024-04-22T07:13:44.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3481 16 3599 3583 0.0643221 0 - 0.36018 2024-04-22T07:13:45.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3482 16 3599 3583 0.0643037 0 - 0.36018 2024-04-22T07:13:46.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3483 16 3599 3583 0.0642852 0 - 0.36018 2024-04-22T07:13:47.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3484 16 3599 3583 0.0642667 0 - 0.36018 2024-04-22T07:13:48.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3485 16 3599 3583 0.0642483 0 - 0.36018 2024-04-22T07:13:49.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3486 16 3599 3583 0.0642299 0 - 0.36018 2024-04-22T07:13:50.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3487 16 3599 3583 0.0642114 0 - 0.36018 2024-04-22T07:13:51.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3488 16 3599 3583 0.064193 0 - 0.36018 2024-04-22T07:13:52.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3489 16 3599 3583 0.0641746 0 - 0.36018 2024-04-22T07:13:53.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3490 16 3599 3583 0.0641563 0 - 0.36018 2024-04-22T07:13:54.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3491 16 3599 3583 0.0641379 0 - 0.36018 2024-04-22T07:13:55.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3492 16 3599 3583 0.0641195 0 - 0.36018 2024-04-22T07:13:56.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3493 16 3599 3583 0.0641012 0 - 0.36018 2024-04-22T07:13:57.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3494 16 3599 3583 0.0640828 0 - 0.36018 2024-04-22T07:13:58.948 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3495 16 3599 3583 0.0640645 0 - 0.36018 2024-04-22T07:13:59.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3496 16 3599 3583 0.0640461 0 - 0.36018 2024-04-22T07:14:00.839 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:14:00.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:14:00.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3497 16 3599 3583 0.0640278 0 - 0.36018 2024-04-22T07:14:01.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3498 16 3599 3583 0.0640095 0 - 0.36018 2024-04-22T07:14:02.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:14:02.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3499 16 3599 3583 0.0639912 0 - 0.36018 2024-04-22T07:14:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:14:02.964+0000 7fa66646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:14:03.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:14:03.952711+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:14:03.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:14:03.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3500 16 3599 3583 0.0639729 0 - 0.36018 2024-04-22T07:14:04.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3501 16 3599 3583 0.0639547 0 - 0.36018 2024-04-22T07:14:05.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3502 16 3599 3583 0.0639364 0 - 0.36018 2024-04-22T07:14:06.949 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3503 16 3599 3583 0.0639182 0 - 0.36018 2024-04-22T07:14:07.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3504 16 3599 3583 0.0638999 0 - 0.36018 2024-04-22T07:14:08.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3505 16 3599 3583 0.0638817 0 - 0.36018 2024-04-22T07:14:09.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3506 16 3599 3583 0.0638635 0 - 0.36018 2024-04-22T07:14:10.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3507 16 3599 3583 0.0638453 0 - 0.36018 2024-04-22T07:14:11.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3508 16 3599 3583 0.0638271 0 - 0.36018 2024-04-22T07:14:12.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3509 16 3599 3583 0.0638089 0 - 0.36018 2024-04-22T07:14:13.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3510 16 3599 3583 0.0637907 0 - 0.36018 2024-04-22T07:14:14.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3511 16 3599 3583 0.0637725 0 - 0.36018 2024-04-22T07:14:15.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3512 16 3599 3583 0.0637544 0 - 0.36018 2024-04-22T07:14:16.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3513 16 3599 3583 0.0637362 0 - 0.36018 2024-04-22T07:14:17.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3514 16 3599 3583 0.0637181 0 - 0.36018 2024-04-22T07:14:18.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3515 16 3599 3583 0.0636999 0 - 0.36018 2024-04-22T07:14:19.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3516 16 3599 3583 0.0636818 0 - 0.36018 2024-04-22T07:14:20.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3517 16 3599 3583 0.0636637 0 - 0.36018 2024-04-22T07:14:21.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3518 16 3599 3583 0.0636456 0 - 0.36018 2024-04-22T07:14:22.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3519 16 3599 3583 0.0636275 0 - 0.36018 2024-04-22T07:14:23.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:14:23.955667+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:14:23.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:14:23.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3520 16 3599 3583 0.0636095 0 - 0.36018 2024-04-22T07:14:24.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3521 16 3599 3583 0.0635914 0 - 0.36018 2024-04-22T07:14:25.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3522 16 3599 3583 0.0635733 0 - 0.36018 2024-04-22T07:14:26.952 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3523 16 3599 3583 0.0635553 0 - 0.36018 2024-04-22T07:14:27.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3524 16 3599 3583 0.0635373 0 - 0.36018 2024-04-22T07:14:28.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3525 16 3599 3583 0.0635192 0 - 0.36018 2024-04-22T07:14:29.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3526 16 3599 3583 0.0635012 0 - 0.36018 2024-04-22T07:14:30.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3527 16 3599 3583 0.0634832 0 - 0.36018 2024-04-22T07:14:31.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3528 16 3599 3583 0.0634652 0 - 0.36018 2024-04-22T07:14:32.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3529 16 3599 3583 0.0634472 0 - 0.36018 2024-04-22T07:14:33.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3530 16 3599 3583 0.0634293 0 - 0.36018 2024-04-22T07:14:34.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3531 16 3599 3583 0.0634113 0 - 0.36018 2024-04-22T07:14:35.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3532 16 3599 3583 0.0633934 0 - 0.36018 2024-04-22T07:14:36.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3533 16 3599 3583 0.0633754 0 - 0.36018 2024-04-22T07:14:37.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3534 16 3599 3583 0.0633575 0 - 0.36018 2024-04-22T07:14:38.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3535 16 3599 3583 0.0633396 0 - 0.36018 2024-04-22T07:14:39.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3536 16 3599 3583 0.0633216 0 - 0.36018 2024-04-22T07:14:40.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3537 16 3599 3583 0.0633037 0 - 0.36018 2024-04-22T07:14:41.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3538 16 3599 3583 0.0632858 0 - 0.36018 2024-04-22T07:14:42.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3539 16 3599 3583 0.063268 0 - 0.36018 2024-04-22T07:14:43.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:14:43.958439+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:14:43.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:14:43.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3540 16 3599 3583 0.0632501 0 - 0.36018 2024-04-22T07:14:44.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3541 16 3599 3583 0.0632322 0 - 0.36018 2024-04-22T07:14:45.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3542 16 3599 3583 0.0632144 0 - 0.36018 2024-04-22T07:14:46.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3543 16 3599 3583 0.0631965 0 - 0.36018 2024-04-22T07:14:47.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3544 16 3599 3583 0.0631787 0 - 0.36018 2024-04-22T07:14:48.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3545 16 3599 3583 0.0631609 0 - 0.36018 2024-04-22T07:14:49.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3546 16 3599 3583 0.0631431 0 - 0.36018 2024-04-22T07:14:50.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3547 16 3599 3583 0.0631253 0 - 0.36018 2024-04-22T07:14:51.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3548 16 3599 3583 0.0631075 0 - 0.36018 2024-04-22T07:14:52.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3549 16 3599 3583 0.0630897 0 - 0.36018 2024-04-22T07:14:53.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3550 16 3599 3583 0.0630719 0 - 0.36018 2024-04-22T07:14:54.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3551 16 3599 3583 0.0630542 0 - 0.36018 2024-04-22T07:14:55.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3552 16 3599 3583 0.0630364 0 - 0.36018 2024-04-22T07:14:56.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3553 16 3599 3583 0.0630187 0 - 0.36018 2024-04-22T07:14:57.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3554 16 3599 3583 0.0630009 0 - 0.36018 2024-04-22T07:14:58.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3555 16 3599 3583 0.0629832 0 - 0.36018 2024-04-22T07:14:59.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3556 16 3599 3583 0.0629655 0 - 0.36018 2024-04-22T07:15:00.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3557 16 3599 3583 0.0629478 0 - 0.36018 2024-04-22T07:15:01.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3558 16 3599 3583 0.0629301 0 - 0.36018 2024-04-22T07:15:02.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3559 16 3599 3583 0.0629124 0 - 0.36018 2024-04-22T07:15:03.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:15:03.961297+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:15:03.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:15:03.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3560 16 3599 3583 0.0628948 0 - 0.36018 2024-04-22T07:15:04.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3561 16 3599 3583 0.0628771 0 - 0.36018 2024-04-22T07:15:05.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3562 16 3599 3583 0.0628594 0 - 0.36018 2024-04-22T07:15:06.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3563 16 3599 3583 0.0628418 0 - 0.36018 2024-04-22T07:15:07.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3564 16 3599 3583 0.0628242 0 - 0.36018 2024-04-22T07:15:08.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3565 16 3599 3583 0.0628065 0 - 0.36018 2024-04-22T07:15:09.958 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3566 16 3599 3583 0.0627889 0 - 0.36018 2024-04-22T07:15:10.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3567 16 3599 3583 0.0627713 0 - 0.36018 2024-04-22T07:15:11.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3568 16 3599 3583 0.0627537 0 - 0.36018 2024-04-22T07:15:12.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3569 16 3599 3583 0.0627362 0 - 0.36018 2024-04-22T07:15:13.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3570 16 3599 3583 0.0627186 0 - 0.36018 2024-04-22T07:15:14.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3571 16 3599 3583 0.062701 0 - 0.36018 2024-04-22T07:15:15.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3572 16 3599 3583 0.0626835 0 - 0.36018 2024-04-22T07:15:16.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3573 16 3599 3583 0.0626659 0 - 0.36018 2024-04-22T07:15:17.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3574 16 3599 3583 0.0626484 0 - 0.36018 2024-04-22T07:15:18.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3575 16 3599 3583 0.0626309 0 - 0.36018 2024-04-22T07:15:19.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3576 16 3599 3583 0.0626133 0 - 0.36018 2024-04-22T07:15:20.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3577 16 3599 3583 0.0625958 0 - 0.36018 2024-04-22T07:15:21.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3578 16 3599 3583 0.0625783 0 - 0.36018 2024-04-22T07:15:22.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3579 16 3599 3583 0.0625609 0 - 0.36018 2024-04-22T07:15:23.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:15:23.964225+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:15:23.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:15:23.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3580 16 3599 3583 0.0625434 0 - 0.36018 2024-04-22T07:15:24.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3581 16 3599 3583 0.0625259 0 - 0.36018 2024-04-22T07:15:25.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3582 16 3599 3583 0.0625085 0 - 0.36018 2024-04-22T07:15:26.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3583 16 3599 3583 0.062491 0 - 0.36018 2024-04-22T07:15:27.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3584 16 3599 3583 0.0624736 0 - 0.36018 2024-04-22T07:15:28.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3585 16 3599 3583 0.0624562 0 - 0.36018 2024-04-22T07:15:29.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3586 16 3599 3583 0.0624387 0 - 0.36018 2024-04-22T07:15:30.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3587 16 3599 3583 0.0624213 0 - 0.36018 2024-04-22T07:15:31.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3588 16 3599 3583 0.0624039 0 - 0.36018 2024-04-22T07:15:32.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3589 16 3599 3583 0.0623865 0 - 0.36018 2024-04-22T07:15:33.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3590 16 3599 3583 0.0623692 0 - 0.36018 2024-04-22T07:15:34.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3591 16 3599 3583 0.0623518 0 - 0.36018 2024-04-22T07:15:35.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3592 16 3599 3583 0.0623344 0 - 0.36018 2024-04-22T07:15:36.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3593 16 3599 3583 0.0623171 0 - 0.36018 2024-04-22T07:15:37.963 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3594 16 3599 3583 0.0622998 0 - 0.36018 2024-04-22T07:15:38.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3595 16 3599 3583 0.0622824 0 - 0.36018 2024-04-22T07:15:39.963 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3596 16 3599 3583 0.0622651 0 - 0.36018 2024-04-22T07:15:40.963 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3597 16 3599 3583 0.0622478 0 - 0.36018 2024-04-22T07:15:41.963 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3598 16 3599 3583 0.0622305 0 - 0.36018 2024-04-22T07:15:42.963 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3599 16 3599 3583 0.0622132 0 - 0.36018 2024-04-22T07:15:43.963 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:15:43.967062+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:15:43.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:15:43.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3600 16 3599 3583 0.0621959 0 - 0.36018 2024-04-22T07:15:44.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3601 16 3599 3583 0.0621787 0 - 0.36018 2024-04-22T07:15:45.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3602 16 3599 3583 0.0621614 0 - 0.36018 2024-04-22T07:15:46.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3603 16 3599 3583 0.0621441 0 - 0.36018 2024-04-22T07:15:47.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3604 16 3599 3583 0.0621269 0 - 0.36018 2024-04-22T07:15:48.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3605 16 3599 3583 0.0621097 0 - 0.36018 2024-04-22T07:15:49.964 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3606 16 3599 3583 0.0620924 0 - 0.36018 2024-04-22T07:15:50.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3607 16 3599 3583 0.0620752 0 - 0.36018 2024-04-22T07:15:51.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3608 16 3599 3583 0.062058 0 - 0.36018 2024-04-22T07:15:52.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3609 16 3599 3583 0.0620408 0 - 0.36018 2024-04-22T07:15:53.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3610 16 3599 3583 0.0620236 0 - 0.36018 2024-04-22T07:15:54.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3611 16 3599 3583 0.0620065 0 - 0.36018 2024-04-22T07:15:55.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3612 16 3599 3583 0.0619893 0 - 0.36018 2024-04-22T07:15:56.965 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3613 16 3599 3583 0.0619721 0 - 0.36018 2024-04-22T07:15:57.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3614 16 3599 3583 0.061955 0 - 0.36018 2024-04-22T07:15:58.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3615 16 3599 3583 0.0619378 0 - 0.36018 2024-04-22T07:15:59.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3616 16 3599 3583 0.0619207 0 - 0.36018 2024-04-22T07:16:00.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3617 16 3599 3583 0.0619036 0 - 0.36018 2024-04-22T07:16:01.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3618 16 3599 3583 0.0618865 0 - 0.36018 2024-04-22T07:16:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:16:02.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:16:02.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3619 16 3599 3583 0.0618694 0 - 0.36018 2024-04-22T07:16:03.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:16:03.969982+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:16:03.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:16:03.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3620 16 3599 3583 0.0618523 0 - 0.36018 2024-04-22T07:16:04.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:16:04.966 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3621 16 3599 3583 0.0618352 0 - 0.36018 2024-04-22T07:16:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:16:05.001+0000 7f256435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:16:05.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3622 16 3599 3583 0.0618181 0 - 0.36018 2024-04-22T07:16:06.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3623 16 3599 3583 0.0618011 0 - 0.36018 2024-04-22T07:16:07.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3624 16 3599 3583 0.061784 0 - 0.36018 2024-04-22T07:16:08.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3625 16 3599 3583 0.061767 0 - 0.36018 2024-04-22T07:16:09.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3626 16 3599 3583 0.0617499 0 - 0.36018 2024-04-22T07:16:10.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3627 16 3599 3583 0.0617329 0 - 0.36018 2024-04-22T07:16:11.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3628 16 3599 3583 0.0617159 0 - 0.36018 2024-04-22T07:16:12.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3629 16 3599 3583 0.0616989 0 - 0.36018 2024-04-22T07:16:13.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3630 16 3599 3583 0.0616819 0 - 0.36018 2024-04-22T07:16:14.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3631 16 3599 3583 0.0616649 0 - 0.36018 2024-04-22T07:16:15.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3632 16 3599 3583 0.0616479 0 - 0.36018 2024-04-22T07:16:16.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3633 16 3599 3583 0.061631 0 - 0.36018 2024-04-22T07:16:17.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3634 16 3599 3583 0.061614 0 - 0.36018 2024-04-22T07:16:18.968 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3635 16 3599 3583 0.0615971 0 - 0.36018 2024-04-22T07:16:19.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3636 16 3599 3583 0.0615801 0 - 0.36018 2024-04-22T07:16:20.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3637 16 3599 3583 0.0615632 0 - 0.36018 2024-04-22T07:16:21.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3638 16 3599 3583 0.0615463 0 - 0.36018 2024-04-22T07:16:22.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3639 16 3599 3583 0.0615294 0 - 0.36018 2024-04-22T07:16:23.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:16:23.972877+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:16:23.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:16:23.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3640 16 3599 3583 0.0615124 0 - 0.36018 2024-04-22T07:16:24.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3641 16 3599 3583 0.0614956 0 - 0.36018 2024-04-22T07:16:25.969 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3642 16 3599 3583 0.0614787 0 - 0.36018 2024-04-22T07:16:26.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3643 16 3599 3583 0.0614618 0 - 0.36018 2024-04-22T07:16:27.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3644 16 3599 3583 0.0614449 0 - 0.36018 2024-04-22T07:16:28.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3645 16 3599 3583 0.0614281 0 - 0.36018 2024-04-22T07:16:29.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3646 16 3599 3583 0.0614112 0 - 0.36018 2024-04-22T07:16:30.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3647 16 3599 3583 0.0613944 0 - 0.36018 2024-04-22T07:16:31.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3648 16 3599 3583 0.0613776 0 - 0.36018 2024-04-22T07:16:32.970 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3649 16 3599 3583 0.0613607 0 - 0.36018 2024-04-22T07:16:33.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3650 16 3599 3583 0.0613439 0 - 0.36018 2024-04-22T07:16:34.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3651 16 3599 3583 0.0613271 0 - 0.36018 2024-04-22T07:16:35.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3652 16 3599 3583 0.0613103 0 - 0.36018 2024-04-22T07:16:36.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3653 16 3599 3583 0.0612935 0 - 0.36018 2024-04-22T07:16:37.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3654 16 3599 3583 0.0612768 0 - 0.36018 2024-04-22T07:16:38.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3655 16 3599 3583 0.06126 0 - 0.36018 2024-04-22T07:16:39.971 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3656 16 3599 3583 0.0612432 0 - 0.36018 2024-04-22T07:16:40.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3657 16 3599 3583 0.0612265 0 - 0.36018 2024-04-22T07:16:41.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3658 16 3599 3583 0.0612098 0 - 0.36018 2024-04-22T07:16:42.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3659 16 3599 3583 0.061193 0 - 0.36018 2024-04-22T07:16:43.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:16:43.975674+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:16:43.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:16:43.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3660 16 3599 3583 0.0611763 0 - 0.36018 2024-04-22T07:16:44.972 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3661 16 3599 3583 0.0611596 0 - 0.36018 2024-04-22T07:16:45.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3662 16 3599 3583 0.0611429 0 - 0.36018 2024-04-22T07:16:46.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3663 16 3599 3583 0.0611262 0 - 0.36018 2024-04-22T07:16:47.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3664 16 3599 3583 0.0611095 0 - 0.36018 2024-04-22T07:16:48.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3665 16 3599 3583 0.0610929 0 - 0.36018 2024-04-22T07:16:49.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3666 16 3599 3583 0.0610762 0 - 0.36018 2024-04-22T07:16:50.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3667 16 3599 3583 0.0610595 0 - 0.36018 2024-04-22T07:16:51.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3668 16 3599 3583 0.0610429 0 - 0.36018 2024-04-22T07:16:52.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3669 16 3599 3583 0.0610263 0 - 0.36018 2024-04-22T07:16:53.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3670 16 3599 3583 0.0610096 0 - 0.36018 2024-04-22T07:16:54.973 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3671 16 3599 3583 0.060993 0 - 0.36018 2024-04-22T07:16:55.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3672 16 3599 3583 0.0609764 0 - 0.36018 2024-04-22T07:16:56.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3673 16 3599 3583 0.0609598 0 - 0.36018 2024-04-22T07:16:57.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3674 16 3599 3583 0.0609432 0 - 0.36018 2024-04-22T07:16:58.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3675 16 3599 3583 0.0609266 0 - 0.36018 2024-04-22T07:16:59.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3676 16 3599 3583 0.06091 0 - 0.36018 2024-04-22T07:17:00.974 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3677 16 3599 3583 0.0608935 0 - 0.36018 2024-04-22T07:17:01.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3678 16 3599 3583 0.0608769 0 - 0.36018 2024-04-22T07:17:02.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3679 16 3599 3583 0.0608604 0 - 0.36018 2024-04-22T07:17:03.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:17:03.978648+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:17:03.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:17:03.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3680 16 3599 3583 0.0608438 0 - 0.36018 2024-04-22T07:17:04.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3681 16 3599 3583 0.0608273 0 - 0.36018 2024-04-22T07:17:05.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3682 16 3599 3583 0.0608108 0 - 0.36018 2024-04-22T07:17:06.975 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3683 16 3599 3583 0.0607943 0 - 0.36018 2024-04-22T07:17:07.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3684 16 3599 3583 0.0607778 0 - 0.36018 2024-04-22T07:17:08.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3685 16 3599 3583 0.0607613 0 - 0.36018 2024-04-22T07:17:09.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3686 16 3599 3583 0.0607448 0 - 0.36018 2024-04-22T07:17:10.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3687 16 3599 3583 0.0607283 0 - 0.36018 2024-04-22T07:17:11.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3688 16 3599 3583 0.0607119 0 - 0.36018 2024-04-22T07:17:12.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3689 16 3599 3583 0.0606954 0 - 0.36018 2024-04-22T07:17:13.976 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3690 16 3599 3583 0.0606789 0 - 0.36018 2024-04-22T07:17:14.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3691 16 3599 3583 0.0606625 0 - 0.36018 2024-04-22T07:17:15.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3692 16 3599 3583 0.0606461 0 - 0.36018 2024-04-22T07:17:16.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3693 16 3599 3583 0.0606297 0 - 0.36018 2024-04-22T07:17:17.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3694 16 3599 3583 0.0606132 0 - 0.36018 2024-04-22T07:17:18.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3695 16 3599 3583 0.0605968 0 - 0.36018 2024-04-22T07:17:19.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3696 16 3599 3583 0.0605804 0 - 0.36018 2024-04-22T07:17:20.977 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3697 16 3599 3583 0.0605641 0 - 0.36018 2024-04-22T07:17:21.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3698 16 3599 3583 0.0605477 0 - 0.36018 2024-04-22T07:17:22.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3699 16 3599 3583 0.0605313 0 - 0.36018 2024-04-22T07:17:23.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:17:23.981471+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:17:23.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:17:23.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3700 16 3599 3583 0.060515 0 - 0.36018 2024-04-22T07:17:24.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3701 16 3599 3583 0.0604986 0 - 0.36018 2024-04-22T07:17:25.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3702 16 3599 3583 0.0604823 0 - 0.36018 2024-04-22T07:17:26.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3703 16 3599 3583 0.0604659 0 - 0.36018 2024-04-22T07:17:27.978 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3704 16 3599 3583 0.0604496 0 - 0.36018 2024-04-22T07:17:28.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3705 16 3599 3583 0.0604333 0 - 0.36018 2024-04-22T07:17:29.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3706 16 3599 3583 0.060417 0 - 0.36018 2024-04-22T07:17:30.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3707 16 3599 3583 0.0604007 0 - 0.36018 2024-04-22T07:17:31.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3708 16 3599 3583 0.0603844 0 - 0.36018 2024-04-22T07:17:32.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3709 16 3599 3583 0.0603681 0 - 0.36018 2024-04-22T07:17:33.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3710 16 3599 3583 0.0603518 0 - 0.36018 2024-04-22T07:17:34.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3711 16 3599 3583 0.0603356 0 - 0.36018 2024-04-22T07:17:35.979 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3712 16 3599 3583 0.0603193 0 - 0.36018 2024-04-22T07:17:36.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3713 16 3599 3583 0.0603031 0 - 0.36018 2024-04-22T07:17:37.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3714 16 3599 3583 0.0602868 0 - 0.36018 2024-04-22T07:17:38.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3715 16 3599 3583 0.0602706 0 - 0.36018 2024-04-22T07:17:39.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3716 16 3599 3583 0.0602544 0 - 0.36018 2024-04-22T07:17:40.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3717 16 3599 3583 0.0602382 0 - 0.36018 2024-04-22T07:17:41.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3718 16 3599 3583 0.060222 0 - 0.36018 2024-04-22T07:17:42.980 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3719 16 3599 3583 0.0602058 0 - 0.36018 2024-04-22T07:17:43.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:17:43.984302+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:17:43.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:17:43.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3720 16 3599 3583 0.0601896 0 - 0.36018 2024-04-22T07:17:44.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3721 16 3599 3583 0.0601734 0 - 0.36018 2024-04-22T07:17:45.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3722 16 3599 3583 0.0601573 0 - 0.36018 2024-04-22T07:17:46.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3723 16 3599 3583 0.0601411 0 - 0.36018 2024-04-22T07:17:47.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3724 16 3599 3583 0.060125 0 - 0.36018 2024-04-22T07:17:48.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3725 16 3599 3583 0.0601088 0 - 0.36018 2024-04-22T07:17:49.981 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3726 16 3599 3583 0.0600927 0 - 0.36018 2024-04-22T07:17:50.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3727 16 3599 3583 0.0600766 0 - 0.36018 2024-04-22T07:17:51.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3728 16 3599 3583 0.0600604 0 - 0.36018 2024-04-22T07:17:52.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3729 16 3599 3583 0.0600443 0 - 0.36018 2024-04-22T07:17:53.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3730 16 3599 3583 0.0600282 0 - 0.36018 2024-04-22T07:17:54.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3731 16 3599 3583 0.0600121 0 - 0.36018 2024-04-22T07:17:55.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3732 16 3599 3583 0.0599961 0 - 0.36018 2024-04-22T07:17:56.982 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3733 16 3599 3583 0.05998 0 - 0.36018 2024-04-22T07:17:57.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3734 16 3599 3583 0.0599639 0 - 0.36018 2024-04-22T07:17:58.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3735 16 3599 3583 0.0599479 0 - 0.36018 2024-04-22T07:17:59.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3736 16 3599 3583 0.0599318 0 - 0.36018 2024-04-22T07:18:00.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3737 16 3599 3583 0.0599158 0 - 0.36018 2024-04-22T07:18:01.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3738 16 3599 3583 0.0598998 0 - 0.36018 2024-04-22T07:18:02.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3739 16 3599 3583 0.0598837 0 - 0.36018 2024-04-22T07:18:03.983 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:18:03.987102+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:18:03.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:18:03.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3740 16 3599 3583 0.0598677 0 - 0.36018 2024-04-22T07:18:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:18:04.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:18:04.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3741 16 3599 3583 0.0598517 0 - 0.36018 2024-04-22T07:18:05.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3742 16 3599 3583 0.0598357 0 - 0.36018 2024-04-22T07:18:06.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:18:06.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3743 16 3599 3583 0.0598198 0 - 0.36018 2024-04-22T07:18:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:18:07.038+0000 7f7e5a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:18:07.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3744 16 3599 3583 0.0598038 0 - 0.36018 2024-04-22T07:18:08.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3745 16 3599 3583 0.0597878 0 - 0.36018 2024-04-22T07:18:09.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3746 16 3599 3583 0.0597718 0 - 0.36018 2024-04-22T07:18:10.984 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3747 16 3599 3583 0.0597559 0 - 0.36018 2024-04-22T07:18:11.985 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3748 16 3599 3583 0.0597399 0 - 0.36018 2024-04-22T07:18:12.985 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3749 16 3599 3583 0.059724 0 - 0.36018 2024-04-22T07:18:13.985 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3750 16 3599 3583 0.0597081 0 - 0.36018 2024-04-22T07:18:14.985 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3751 16 3599 3583 0.0596922 0 - 0.36018 2024-04-22T07:18:15.985 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3752 16 3599 3583 0.0596763 0 - 0.36018 2024-04-22T07:18:16.985 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3753 16 3599 3583 0.0596604 0 - 0.36018 2024-04-22T07:18:17.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3754 16 3599 3583 0.0596445 0 - 0.36018 2024-04-22T07:18:18.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3755 16 3599 3583 0.0596286 0 - 0.36018 2024-04-22T07:18:19.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3756 16 3599 3583 0.0596127 0 - 0.36018 2024-04-22T07:18:20.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3757 16 3599 3583 0.0595968 0 - 0.36018 2024-04-22T07:18:21.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3758 16 3599 3583 0.059581 0 - 0.36018 2024-04-22T07:18:22.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3759 16 3599 3583 0.0595651 0 - 0.36018 2024-04-22T07:18:23.986 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:18:23.990126+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:18:23.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:18:23.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3760 16 3599 3583 0.0595493 0 - 0.36018 2024-04-22T07:18:24.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3761 16 3599 3583 0.0595335 0 - 0.36018 2024-04-22T07:18:25.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3762 16 3599 3583 0.0595176 0 - 0.36018 2024-04-22T07:18:26.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3763 16 3599 3583 0.0595018 0 - 0.36018 2024-04-22T07:18:27.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3764 16 3599 3583 0.059486 0 - 0.36018 2024-04-22T07:18:28.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3765 16 3599 3583 0.0594702 0 - 0.36018 2024-04-22T07:18:29.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3766 16 3599 3583 0.0594544 0 - 0.36018 2024-04-22T07:18:30.987 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3767 16 3599 3583 0.0594386 0 - 0.36018 2024-04-22T07:18:31.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3768 16 3599 3583 0.0594229 0 - 0.36018 2024-04-22T07:18:32.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3769 16 3599 3583 0.0594071 0 - 0.36018 2024-04-22T07:18:33.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3770 16 3599 3583 0.0593913 0 - 0.36018 2024-04-22T07:18:34.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3771 16 3599 3583 0.0593756 0 - 0.36018 2024-04-22T07:18:35.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3772 16 3599 3583 0.0593598 0 - 0.36018 2024-04-22T07:18:36.988 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3773 16 3599 3583 0.0593441 0 - 0.36018 2024-04-22T07:18:37.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3774 16 3599 3583 0.0593284 0 - 0.36018 2024-04-22T07:18:38.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3775 16 3599 3583 0.0593127 0 - 0.36018 2024-04-22T07:18:39.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3776 16 3599 3583 0.059297 0 - 0.36018 2024-04-22T07:18:40.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3777 16 3599 3583 0.0592813 0 - 0.36018 2024-04-22T07:18:41.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3778 16 3599 3583 0.0592656 0 - 0.36018 2024-04-22T07:18:42.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3779 16 3599 3583 0.0592499 0 - 0.36018 2024-04-22T07:18:43.989 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:18:43.993083+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:18:43.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:18:43.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3780 16 3599 3583 0.0592342 0 - 0.36018 2024-04-22T07:18:44.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3781 16 3599 3583 0.0592185 0 - 0.36018 2024-04-22T07:18:45.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3782 16 3599 3583 0.0592029 0 - 0.36018 2024-04-22T07:18:46.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3783 16 3599 3583 0.0591872 0 - 0.36018 2024-04-22T07:18:47.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3784 16 3599 3583 0.0591716 0 - 0.36018 2024-04-22T07:18:48.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3785 16 3599 3583 0.059156 0 - 0.36018 2024-04-22T07:18:49.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3786 16 3599 3583 0.0591403 0 - 0.36018 2024-04-22T07:18:50.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3787 16 3599 3583 0.0591247 0 - 0.36018 2024-04-22T07:18:51.990 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3788 16 3599 3583 0.0591091 0 - 0.36018 2024-04-22T07:18:52.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3789 16 3599 3583 0.0590935 0 - 0.36018 2024-04-22T07:18:53.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3790 16 3599 3583 0.0590779 0 - 0.36018 2024-04-22T07:18:54.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3791 16 3599 3583 0.0590623 0 - 0.36018 2024-04-22T07:18:55.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3792 16 3599 3583 0.0590468 0 - 0.36018 2024-04-22T07:18:56.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3793 16 3599 3583 0.0590312 0 - 0.36018 2024-04-22T07:18:57.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3794 16 3599 3583 0.0590156 0 - 0.36018 2024-04-22T07:18:58.991 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3795 16 3599 3583 0.0590001 0 - 0.36018 2024-04-22T07:18:59.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3796 16 3599 3583 0.0589845 0 - 0.36018 2024-04-22T07:19:00.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3797 16 3599 3583 0.058969 0 - 0.36018 2024-04-22T07:19:01.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3798 16 3599 3583 0.0589535 0 - 0.36018 2024-04-22T07:19:02.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3799 16 3599 3583 0.058938 0 - 0.36018 2024-04-22T07:19:03.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:19:03.995979+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:19:03.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:19:03.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3800 16 3599 3583 0.0589225 0 - 0.36018 2024-04-22T07:19:04.992 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3801 16 3599 3583 0.0589069 0 - 0.36018 2024-04-22T07:19:05.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3802 16 3599 3583 0.0588915 0 - 0.36018 2024-04-22T07:19:06.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3803 16 3599 3583 0.058876 0 - 0.36018 2024-04-22T07:19:07.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3804 16 3599 3583 0.0588605 0 - 0.36018 2024-04-22T07:19:08.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3805 16 3599 3583 0.058845 0 - 0.36018 2024-04-22T07:19:09.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3806 16 3599 3583 0.0588296 0 - 0.36018 2024-04-22T07:19:10.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3807 16 3599 3583 0.0588141 0 - 0.36018 2024-04-22T07:19:11.993 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3808 16 3599 3583 0.0587987 0 - 0.36018 2024-04-22T07:19:12.994 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3809 16 3599 3583 0.0587832 0 - 0.36018 2024-04-22T07:19:13.994 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3810 16 3599 3583 0.0587678 0 - 0.36018 2024-04-22T07:19:14.994 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3811 16 3599 3583 0.0587524 0 - 0.36018 2024-04-22T07:19:15.994 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3812 16 3599 3583 0.058737 0 - 0.36018 2024-04-22T07:19:16.994 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3813 16 3599 3583 0.0587216 0 - 0.36018 2024-04-22T07:19:17.994 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3814 16 3599 3583 0.0587062 0 - 0.36018 2024-04-22T07:19:18.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3815 16 3599 3583 0.0586908 0 - 0.36018 2024-04-22T07:19:19.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3816 16 3599 3583 0.0586754 0 - 0.36018 2024-04-22T07:19:20.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3817 16 3599 3583 0.05866 0 - 0.36018 2024-04-22T07:19:21.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3818 16 3599 3583 0.0586447 0 - 0.36018 2024-04-22T07:19:22.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3819 16 3599 3583 0.0586293 0 - 0.36018 2024-04-22T07:19:23.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:19:23.999003+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:19:23.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:19:23.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3820 16 3599 3583 0.058614 0 - 0.36018 2024-04-22T07:19:24.995 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3821 16 3599 3583 0.0585986 0 - 0.36018 2024-04-22T07:19:25.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3822 16 3599 3583 0.0585833 0 - 0.36018 2024-04-22T07:19:26.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3823 16 3599 3583 0.058568 0 - 0.36018 2024-04-22T07:19:27.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3824 16 3599 3583 0.0585526 0 - 0.36018 2024-04-22T07:19:28.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3825 16 3599 3583 0.0585373 0 - 0.36018 2024-04-22T07:19:29.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3826 16 3599 3583 0.058522 0 - 0.36018 2024-04-22T07:19:30.996 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3827 16 3599 3583 0.0585067 0 - 0.36018 2024-04-22T07:19:31.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3828 16 3599 3583 0.0584915 0 - 0.36018 2024-04-22T07:19:32.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3829 16 3599 3583 0.0584762 0 - 0.36018 2024-04-22T07:19:33.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3830 16 3599 3583 0.0584609 0 - 0.36018 2024-04-22T07:19:34.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3831 16 3599 3583 0.0584457 0 - 0.36018 2024-04-22T07:19:35.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3832 16 3599 3583 0.0584304 0 - 0.36018 2024-04-22T07:19:36.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3833 16 3599 3583 0.0584152 0 - 0.36018 2024-04-22T07:19:37.997 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3834 16 3599 3583 0.0583999 0 - 0.36018 2024-04-22T07:19:38.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3835 16 3599 3583 0.0583847 0 - 0.36018 2024-04-22T07:19:39.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3836 16 3599 3583 0.0583695 0 - 0.36018 2024-04-22T07:19:40.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3837 16 3599 3583 0.0583543 0 - 0.36018 2024-04-22T07:19:41.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3838 16 3599 3583 0.0583391 0 - 0.36018 2024-04-22T07:19:42.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3839 16 3599 3583 0.0583239 0 - 0.36018 2024-04-22T07:19:43.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:19:44.001917+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:19:43.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:19:43.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3840 16 3599 3583 0.0583087 0 - 0.36018 2024-04-22T07:19:44.998 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3841 16 3599 3583 0.0582935 0 - 0.36018 2024-04-22T07:19:45.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3842 16 3599 3583 0.0582783 0 - 0.36018 2024-04-22T07:19:46.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3843 16 3599 3583 0.0582632 0 - 0.36018 2024-04-22T07:19:47.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3844 16 3599 3583 0.058248 0 - 0.36018 2024-04-22T07:19:48.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3845 16 3599 3583 0.0582328 0 - 0.36018 2024-04-22T07:19:49.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3846 16 3599 3583 0.0582177 0 - 0.36018 2024-04-22T07:19:50.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3847 16 3599 3583 0.0582026 0 - 0.36018 2024-04-22T07:19:51.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3848 16 3599 3583 0.0581874 0 - 0.36018 2024-04-22T07:19:52.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3849 16 3599 3583 0.0581723 0 - 0.36018 2024-04-22T07:19:53.999 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3850 16 3599 3583 0.0581572 0 - 0.36018 2024-04-22T07:19:55.000 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3851 16 3599 3583 0.0581421 0 - 0.36018 2024-04-22T07:19:56.000 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3852 16 3599 3583 0.058127 0 - 0.36018 2024-04-22T07:19:57.000 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3853 16 3599 3583 0.0581119 0 - 0.36018 2024-04-22T07:19:58.000 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3854 16 3599 3583 0.0580969 0 - 0.36018 2024-04-22T07:19:59.000 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3855 16 3599 3583 0.0580818 0 - 0.36018 2024-04-22T07:20:00.000 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3856 16 3599 3583 0.0580667 0 - 0.36018 2024-04-22T07:20:01.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3857 16 3599 3583 0.0580517 0 - 0.36018 2024-04-22T07:20:02.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3858 16 3599 3583 0.0580366 0 - 0.36018 2024-04-22T07:20:03.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3859 16 3599 3583 0.0580216 0 - 0.36018 2024-04-22T07:20:04.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:20:04.004693+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:20:04.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:20:04.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3860 16 3599 3583 0.0580066 0 - 0.36018 2024-04-22T07:20:05.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3861 16 3599 3583 0.0579915 0 - 0.36018 2024-04-22T07:20:06.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3862 16 3599 3583 0.0579765 0 - 0.36018 2024-04-22T07:20:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:20:06.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:20:07.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3863 16 3599 3583 0.0579615 0 - 0.36018 2024-04-22T07:20:08.001 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3864 16 3599 3583 0.0579465 0 - 0.36018 2024-04-22T07:20:08.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:20:09.002 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3865 16 3599 3583 0.0579315 0 - 0.36018 2024-04-22T07:20:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:20:09.079+0000 7efd79a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:20:10.002 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3866 16 3599 3583 0.0579165 0 - 0.36018 2024-04-22T07:20:11.002 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3867 16 3599 3583 0.0579016 0 - 0.36018 2024-04-22T07:20:12.002 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3868 16 3599 3583 0.0578866 0 - 0.36018 2024-04-22T07:20:13.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3869 16 3599 3583 0.0578716 0 - 0.36018 2024-04-22T07:20:14.002 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3870 16 3599 3583 0.0578567 0 - 0.36018 2024-04-22T07:20:15.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3871 16 3599 3583 0.0578417 0 - 0.36018 2024-04-22T07:20:16.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3872 16 3599 3583 0.0578268 0 - 0.36018 2024-04-22T07:20:17.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3873 16 3599 3583 0.0578119 0 - 0.36018 2024-04-22T07:20:18.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3874 16 3599 3583 0.0577969 0 - 0.36018 2024-04-22T07:20:19.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3875 16 3599 3583 0.057782 0 - 0.36018 2024-04-22T07:20:20.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3876 16 3599 3583 0.0577671 0 - 0.36018 2024-04-22T07:20:21.003 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3877 16 3599 3583 0.0577522 0 - 0.36018 2024-04-22T07:20:22.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3878 16 3599 3583 0.0577373 0 - 0.36018 2024-04-22T07:20:23.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3879 16 3599 3583 0.0577224 0 - 0.36018 2024-04-22T07:20:24.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:20:24.007528+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:20:24.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:20:24.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3880 16 3599 3583 0.0577076 0 - 0.36018 2024-04-22T07:20:25.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3881 16 3599 3583 0.0576927 0 - 0.36018 2024-04-22T07:20:26.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3882 16 3599 3583 0.0576778 0 - 0.36018 2024-04-22T07:20:27.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3883 16 3599 3583 0.057663 0 - 0.36018 2024-04-22T07:20:28.004 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3884 16 3599 3583 0.0576481 0 - 0.36018 2024-04-22T07:20:29.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3885 16 3599 3583 0.0576333 0 - 0.36018 2024-04-22T07:20:30.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3886 16 3599 3583 0.0576185 0 - 0.36018 2024-04-22T07:20:31.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3887 16 3599 3583 0.0576036 0 - 0.36018 2024-04-22T07:20:32.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3888 16 3599 3583 0.0575888 0 - 0.36018 2024-04-22T07:20:33.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3889 16 3599 3583 0.057574 0 - 0.36018 2024-04-22T07:20:34.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3890 16 3599 3583 0.0575592 0 - 0.36018 2024-04-22T07:20:35.005 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3891 16 3599 3583 0.0575444 0 - 0.36018 2024-04-22T07:20:36.006 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3892 16 3599 3583 0.0575296 0 - 0.36018 2024-04-22T07:20:37.006 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3893 16 3599 3583 0.0575148 0 - 0.36018 2024-04-22T07:20:38.006 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3894 16 3599 3583 0.0575001 0 - 0.36018 2024-04-22T07:20:39.006 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3895 16 3599 3583 0.0574853 0 - 0.36018 2024-04-22T07:20:40.006 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3896 16 3599 3583 0.0574706 0 - 0.36018 2024-04-22T07:20:41.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3897 16 3599 3583 0.0574558 0 - 0.36018 2024-04-22T07:20:42.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3898 16 3599 3583 0.0574411 0 - 0.36018 2024-04-22T07:20:43.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3899 16 3599 3583 0.0574263 0 - 0.36018 2024-04-22T07:20:44.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:20:44.010501+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:20:44.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:20:44.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3900 16 3599 3583 0.0574116 0 - 0.36018 2024-04-22T07:20:45.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3901 16 3599 3583 0.0573969 0 - 0.36018 2024-04-22T07:20:46.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3902 16 3599 3583 0.0573822 0 - 0.36018 2024-04-22T07:20:47.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3903 16 3599 3583 0.0573675 0 - 0.36018 2024-04-22T07:20:48.007 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3904 16 3599 3583 0.0573528 0 - 0.36018 2024-04-22T07:20:49.008 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3905 16 3599 3583 0.0573381 0 - 0.36018 2024-04-22T07:20:50.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3906 16 3599 3583 0.0573234 0 - 0.36018 2024-04-22T07:20:51.008 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3907 16 3599 3583 0.0573088 0 - 0.36018 2024-04-22T07:20:52.008 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3908 16 3599 3583 0.0572941 0 - 0.36018 2024-04-22T07:20:53.008 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3909 16 3599 3583 0.0572794 0 - 0.36018 2024-04-22T07:20:54.008 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3910 16 3599 3583 0.0572648 0 - 0.36018 2024-04-22T07:20:55.008 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3911 16 3599 3583 0.0572501 0 - 0.36018 2024-04-22T07:20:56.009 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3912 16 3599 3583 0.0572355 0 - 0.36018 2024-04-22T07:20:57.009 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3913 16 3599 3583 0.0572209 0 - 0.36018 2024-04-22T07:20:58.009 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3914 16 3599 3583 0.0572063 0 - 0.36018 2024-04-22T07:20:59.009 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3915 16 3599 3583 0.0571916 0 - 0.36018 2024-04-22T07:21:00.009 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3916 16 3599 3583 0.057177 0 - 0.36018 2024-04-22T07:21:01.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3917 16 3599 3583 0.0571624 0 - 0.36018 2024-04-22T07:21:02.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3918 16 3599 3583 0.0571479 0 - 0.36018 2024-04-22T07:21:03.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3919 16 3599 3583 0.0571333 0 - 0.36018 2024-04-22T07:21:04.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:21:04.013499+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:21:04.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:21:04.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3920 16 3599 3583 0.0571187 0 - 0.36018 2024-04-22T07:21:05.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3921 16 3599 3583 0.0571041 0 - 0.36018 2024-04-22T07:21:06.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3922 16 3599 3583 0.0570896 0 - 0.36018 2024-04-22T07:21:07.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3923 16 3599 3583 0.057075 0 - 0.36018 2024-04-22T07:21:08.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3924 16 3599 3583 0.0570605 0 - 0.36018 2024-04-22T07:21:09.010 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3925 16 3599 3583 0.0570459 0 - 0.36018 2024-04-22T07:21:10.011 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3926 16 3599 3583 0.0570314 0 - 0.36018 2024-04-22T07:21:11.011 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3927 16 3599 3583 0.0570169 0 - 0.36018 2024-04-22T07:21:12.011 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3928 16 3599 3583 0.0570024 0 - 0.36018 2024-04-22T07:21:13.011 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3929 16 3599 3583 0.0569879 0 - 0.36018 2024-04-22T07:21:14.011 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3930 16 3599 3583 0.0569734 0 - 0.36018 2024-04-22T07:21:15.011 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3931 16 3599 3583 0.0569589 0 - 0.36018 2024-04-22T07:21:16.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3932 16 3599 3583 0.0569444 0 - 0.36018 2024-04-22T07:21:17.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3933 16 3599 3583 0.0569299 0 - 0.36018 2024-04-22T07:21:18.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3934 16 3599 3583 0.0569154 0 - 0.36018 2024-04-22T07:21:19.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3935 16 3599 3583 0.056901 0 - 0.36018 2024-04-22T07:21:20.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3936 16 3599 3583 0.0568865 0 - 0.36018 2024-04-22T07:21:21.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3937 16 3599 3583 0.0568721 0 - 0.36018 2024-04-22T07:21:22.012 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3938 16 3599 3583 0.0568576 0 - 0.36018 2024-04-22T07:21:23.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3939 16 3599 3583 0.0568432 0 - 0.36018 2024-04-22T07:21:24.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:21:24.016422+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:21:24.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:21:24.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3940 16 3599 3583 0.0568288 0 - 0.36018 2024-04-22T07:21:25.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3941 16 3599 3583 0.0568143 0 - 0.36018 2024-04-22T07:21:26.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3942 16 3599 3583 0.0567999 0 - 0.36018 2024-04-22T07:21:27.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3943 16 3599 3583 0.0567855 0 - 0.36018 2024-04-22T07:21:28.013 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3944 16 3599 3583 0.0567711 0 - 0.36018 2024-04-22T07:21:29.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3945 16 3599 3583 0.0567567 0 - 0.36018 2024-04-22T07:21:30.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3946 16 3599 3583 0.0567423 0 - 0.36018 2024-04-22T07:21:31.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3947 16 3599 3583 0.056728 0 - 0.36018 2024-04-22T07:21:32.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3948 16 3599 3583 0.0567136 0 - 0.36018 2024-04-22T07:21:33.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3949 16 3599 3583 0.0566992 0 - 0.36018 2024-04-22T07:21:34.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3950 16 3599 3583 0.0566849 0 - 0.36018 2024-04-22T07:21:35.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3951 16 3599 3583 0.0566705 0 - 0.36018 2024-04-22T07:21:36.014 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3952 16 3599 3583 0.0566562 0 - 0.36018 2024-04-22T07:21:37.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3953 16 3599 3583 0.0566419 0 - 0.36018 2024-04-22T07:21:38.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3954 16 3599 3583 0.0566275 0 - 0.36018 2024-04-22T07:21:39.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3955 16 3599 3583 0.0566132 0 - 0.36018 2024-04-22T07:21:40.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3956 16 3599 3583 0.0565989 0 - 0.36018 2024-04-22T07:21:41.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3957 16 3599 3583 0.0565846 0 - 0.36018 2024-04-22T07:21:42.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3958 16 3599 3583 0.0565703 0 - 0.36018 2024-04-22T07:21:43.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3959 16 3599 3583 0.056556 0 - 0.36018 2024-04-22T07:21:44.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:21:44.019375+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:21:44.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:21:44.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3960 16 3599 3583 0.0565417 0 - 0.36018 2024-04-22T07:21:45.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3961 16 3599 3583 0.0565275 0 - 0.36018 2024-04-22T07:21:46.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3962 16 3599 3583 0.0565132 0 - 0.36018 2024-04-22T07:21:47.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3963 16 3599 3583 0.0564989 0 - 0.36018 2024-04-22T07:21:48.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3964 16 3599 3583 0.0564847 0 - 0.36018 2024-04-22T07:21:49.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3965 16 3599 3583 0.0564704 0 - 0.36018 2024-04-22T07:21:50.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3966 16 3599 3583 0.0564562 0 - 0.36018 2024-04-22T07:21:51.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3967 16 3599 3583 0.056442 0 - 0.36018 2024-04-22T07:21:52.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3968 16 3599 3583 0.0564277 0 - 0.36018 2024-04-22T07:21:53.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3969 16 3599 3583 0.0564135 0 - 0.36018 2024-04-22T07:21:54.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3970 16 3599 3583 0.0563993 0 - 0.36018 2024-04-22T07:21:55.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3971 16 3599 3583 0.0563851 0 - 0.36018 2024-04-22T07:21:56.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3972 16 3599 3583 0.0563709 0 - 0.36018 2024-04-22T07:21:57.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3973 16 3599 3583 0.0563567 0 - 0.36018 2024-04-22T07:21:58.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3974 16 3599 3583 0.0563425 0 - 0.36018 2024-04-22T07:21:59.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3975 16 3599 3583 0.0563284 0 - 0.36018 2024-04-22T07:22:00.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3976 16 3599 3583 0.0563142 0 - 0.36018 2024-04-22T07:22:01.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3977 16 3599 3583 0.0563 0 - 0.36018 2024-04-22T07:22:02.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3978 16 3599 3583 0.0562859 0 - 0.36018 2024-04-22T07:22:03.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3979 16 3599 3583 0.0562717 0 - 0.36018 2024-04-22T07:22:04.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:22:04.022252+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:22:04.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:22:04.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3980 16 3599 3583 0.0562576 0 - 0.36018 2024-04-22T07:22:05.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3981 16 3599 3583 0.0562435 0 - 0.36018 2024-04-22T07:22:06.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3982 16 3599 3583 0.0562294 0 - 0.36018 2024-04-22T07:22:07.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3983 16 3599 3583 0.0562152 0 - 0.36018 2024-04-22T07:22:08.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3984 16 3599 3583 0.0562011 0 - 0.36018 2024-04-22T07:22:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:22:08.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:22:09.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3985 16 3599 3583 0.056187 0 - 0.36018 2024-04-22T07:22:10.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3986 16 3599 3583 0.0561729 0 - 0.36018 2024-04-22T07:22:10.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:22:11.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3987 16 3599 3583 0.0561588 0 - 0.36018 2024-04-22T07:22:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:22:11.115+0000 7fe33d064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:22:12.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3988 16 3599 3583 0.0561448 0 - 0.36018 2024-04-22T07:22:13.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3989 16 3599 3583 0.0561307 0 - 0.36018 2024-04-22T07:22:14.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3990 16 3599 3583 0.0561166 0 - 0.36018 2024-04-22T07:22:15.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3991 16 3599 3583 0.0561026 0 - 0.36018 2024-04-22T07:22:16.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3992 16 3599 3583 0.0560885 0 - 0.36018 2024-04-22T07:22:17.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3993 16 3599 3583 0.0560745 0 - 0.36018 2024-04-22T07:22:18.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3994 16 3599 3583 0.0560604 0 - 0.36018 2024-04-22T07:22:19.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3995 16 3599 3583 0.0560464 0 - 0.36018 2024-04-22T07:22:20.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3996 16 3599 3583 0.0560324 0 - 0.36018 2024-04-22T07:22:21.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3997 16 3599 3583 0.0560183 0 - 0.36018 2024-04-22T07:22:22.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3998 16 3599 3583 0.0560043 0 - 0.36018 2024-04-22T07:22:23.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3999 16 3599 3583 0.0559903 0 - 0.36018 2024-04-22T07:22:24.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:22:24.025185+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:22:24.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:22:24.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4000 16 3599 3583 0.0559763 0 - 0.36018 2024-04-22T07:22:25.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4001 16 3599 3583 0.0559623 0 - 0.36018 2024-04-22T07:22:26.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4002 16 3599 3583 0.0559483 0 - 0.36018 2024-04-22T07:22:27.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4003 16 3599 3583 0.0559344 0 - 0.36018 2024-04-22T07:22:28.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4004 16 3599 3583 0.0559204 0 - 0.36018 2024-04-22T07:22:29.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4005 16 3599 3583 0.0559064 0 - 0.36018 2024-04-22T07:22:30.022 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4006 16 3599 3583 0.0558925 0 - 0.36018 2024-04-22T07:22:31.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4007 16 3599 3583 0.0558785 0 - 0.36018 2024-04-22T07:22:32.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4008 16 3599 3583 0.0558646 0 - 0.36018 2024-04-22T07:22:33.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4009 16 3599 3583 0.0558507 0 - 0.36018 2024-04-22T07:22:34.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4010 16 3599 3583 0.0558367 0 - 0.36018 2024-04-22T07:22:35.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4011 16 3599 3583 0.0558228 0 - 0.36018 2024-04-22T07:22:36.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4012 16 3599 3583 0.0558089 0 - 0.36018 2024-04-22T07:22:37.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4013 16 3599 3583 0.055795 0 - 0.36018 2024-04-22T07:22:38.023 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4014 16 3599 3583 0.0557811 0 - 0.36018 2024-04-22T07:22:39.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4015 16 3599 3583 0.0557672 0 - 0.36018 2024-04-22T07:22:40.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4016 16 3599 3583 0.0557533 0 - 0.36018 2024-04-22T07:22:41.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4017 16 3599 3583 0.0557394 0 - 0.36018 2024-04-22T07:22:42.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4018 16 3599 3583 0.0557256 0 - 0.36018 2024-04-22T07:22:43.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4019 16 3599 3583 0.0557117 0 - 0.36018 2024-04-22T07:22:44.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:22:44.028097+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:22:44.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:22:44.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4020 16 3599 3583 0.0556978 0 - 0.36018 2024-04-22T07:22:45.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4021 16 3599 3583 0.055684 0 - 0.36018 2024-04-22T07:22:46.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4022 16 3599 3583 0.0556701 0 - 0.36018 2024-04-22T07:22:47.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4023 16 3599 3583 0.0556563 0 - 0.36018 2024-04-22T07:22:48.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4024 16 3599 3583 0.0556425 0 - 0.36018 2024-04-22T07:22:49.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4025 16 3599 3583 0.0556286 0 - 0.36018 2024-04-22T07:22:50.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4026 16 3599 3583 0.0556148 0 - 0.36018 2024-04-22T07:22:51.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4027 16 3599 3583 0.055601 0 - 0.36018 2024-04-22T07:22:52.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4028 16 3599 3583 0.0555872 0 - 0.36018 2024-04-22T07:22:53.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4029 16 3599 3583 0.0555734 0 - 0.36018 2024-04-22T07:22:54.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4030 16 3599 3583 0.0555596 0 - 0.36018 2024-04-22T07:22:55.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4031 16 3599 3583 0.0555458 0 - 0.36018 2024-04-22T07:22:56.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4032 16 3599 3583 0.0555321 0 - 0.36018 2024-04-22T07:22:57.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4033 16 3599 3583 0.0555183 0 - 0.36018 2024-04-22T07:22:58.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4034 16 3599 3583 0.0555045 0 - 0.36018 2024-04-22T07:22:59.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4035 16 3599 3583 0.0554908 0 - 0.36018 2024-04-22T07:23:00.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4036 16 3599 3583 0.055477 0 - 0.36018 2024-04-22T07:23:01.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4037 16 3599 3583 0.0554633 0 - 0.36018 2024-04-22T07:23:02.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4038 16 3599 3583 0.0554495 0 - 0.36018 2024-04-22T07:23:03.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4039 16 3599 3583 0.0554358 0 - 0.36018 2024-04-22T07:23:04.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:23:04.031023+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:23:04.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:23:04.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4040 16 3599 3583 0.0554221 0 - 0.36018 2024-04-22T07:23:05.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4041 16 3599 3583 0.0554084 0 - 0.36018 2024-04-22T07:23:06.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4042 16 3599 3583 0.0553947 0 - 0.36018 2024-04-22T07:23:07.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4043 16 3599 3583 0.055381 0 - 0.36018 2024-04-22T07:23:08.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4044 16 3599 3583 0.0553673 0 - 0.36018 2024-04-22T07:23:09.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4045 16 3599 3583 0.0553536 0 - 0.36018 2024-04-22T07:23:10.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4046 16 3599 3583 0.0553399 0 - 0.36018 2024-04-22T07:23:11.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4047 16 3599 3583 0.0553262 0 - 0.36018 2024-04-22T07:23:12.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4048 16 3599 3583 0.0553126 0 - 0.36018 2024-04-22T07:23:13.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4049 16 3599 3583 0.0552989 0 - 0.36018 2024-04-22T07:23:14.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4050 16 3599 3583 0.0552853 0 - 0.36018 2024-04-22T07:23:15.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4051 16 3599 3583 0.0552716 0 - 0.36018 2024-04-22T07:23:16.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4052 16 3599 3583 0.055258 0 - 0.36018 2024-04-22T07:23:17.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4053 16 3599 3583 0.0552443 0 - 0.36018 2024-04-22T07:23:18.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4054 16 3599 3583 0.0552307 0 - 0.36018 2024-04-22T07:23:19.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4055 16 3599 3583 0.0552171 0 - 0.36018 2024-04-22T07:23:20.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4056 16 3599 3583 0.0552035 0 - 0.36018 2024-04-22T07:23:21.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4057 16 3599 3583 0.0551899 0 - 0.36018 2024-04-22T07:23:22.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4058 16 3599 3583 0.0551763 0 - 0.36018 2024-04-22T07:23:23.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4059 16 3599 3583 0.0551627 0 - 0.36018 2024-04-22T07:23:24.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:23:24.033956+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:23:24.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:23:24.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4060 16 3599 3583 0.0551491 0 - 0.36018 2024-04-22T07:23:25.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4061 16 3599 3583 0.0551355 0 - 0.36018 2024-04-22T07:23:26.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4062 16 3599 3583 0.0551219 0 - 0.36018 2024-04-22T07:23:27.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4063 16 3599 3583 0.0551084 0 - 0.36018 2024-04-22T07:23:28.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4064 16 3599 3583 0.0550948 0 - 0.36018 2024-04-22T07:23:29.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4065 16 3599 3583 0.0550812 0 - 0.36018 2024-04-22T07:23:30.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4066 16 3599 3583 0.0550677 0 - 0.36018 2024-04-22T07:23:31.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4067 16 3599 3583 0.0550542 0 - 0.36018 2024-04-22T07:23:32.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4068 16 3599 3583 0.0550406 0 - 0.36018 2024-04-22T07:23:33.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4069 16 3599 3583 0.0550271 0 - 0.36018 2024-04-22T07:23:34.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4070 16 3599 3583 0.0550136 0 - 0.36018 2024-04-22T07:23:35.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4071 16 3599 3583 0.0550001 0 - 0.36018 2024-04-22T07:23:36.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4072 16 3599 3583 0.0549866 0 - 0.36018 2024-04-22T07:23:37.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4073 16 3599 3583 0.0549731 0 - 0.36018 2024-04-22T07:23:38.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4074 16 3599 3583 0.0549596 0 - 0.36018 2024-04-22T07:23:39.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4075 16 3599 3583 0.0549461 0 - 0.36018 2024-04-22T07:23:40.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4076 16 3599 3583 0.0549326 0 - 0.36018 2024-04-22T07:23:41.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4077 16 3599 3583 0.0549191 0 - 0.36018 2024-04-22T07:23:42.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4078 16 3599 3583 0.0549057 0 - 0.36018 2024-04-22T07:23:43.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4079 16 3599 3583 0.0548922 0 - 0.36018 2024-04-22T07:23:44.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:23:44.036911+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:23:44.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:23:44.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4080 16 3599 3583 0.0548787 0 - 0.36018 2024-04-22T07:23:45.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4081 16 3599 3583 0.0548653 0 - 0.36018 2024-04-22T07:23:46.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4082 16 3599 3583 0.0548519 0 - 0.36018 2024-04-22T07:23:47.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4083 16 3599 3583 0.0548384 0 - 0.36018 2024-04-22T07:23:48.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4084 16 3599 3583 0.054825 0 - 0.36018 2024-04-22T07:23:49.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4085 16 3599 3583 0.0548116 0 - 0.36018 2024-04-22T07:23:50.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4086 16 3599 3583 0.0547982 0 - 0.36018 2024-04-22T07:23:51.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4087 16 3599 3583 0.0547847 0 - 0.36018 2024-04-22T07:23:52.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4088 16 3599 3583 0.0547713 0 - 0.36018 2024-04-22T07:23:53.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4089 16 3599 3583 0.054758 0 - 0.36018 2024-04-22T07:23:54.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4090 16 3599 3583 0.0547446 0 - 0.36018 2024-04-22T07:23:55.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4091 16 3599 3583 0.0547312 0 - 0.36018 2024-04-22T07:23:56.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4092 16 3599 3583 0.0547178 0 - 0.36018 2024-04-22T07:23:57.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4093 16 3599 3583 0.0547044 0 - 0.36018 2024-04-22T07:23:58.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4094 16 3599 3583 0.0546911 0 - 0.36018 2024-04-22T07:23:59.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4095 16 3599 3583 0.0546777 0 - 0.36018 2024-04-22T07:24:00.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4096 16 3599 3583 0.0546644 0 - 0.36018 2024-04-22T07:24:01.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4097 16 3599 3583 0.054651 0 - 0.36018 2024-04-22T07:24:02.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4098 16 3599 3583 0.0546377 0 - 0.36018 2024-04-22T07:24:03.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4099 16 3599 3583 0.0546244 0 - 0.36018 2024-04-22T07:24:04.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:24:04.039823+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:24:04.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:24:04.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4100 16 3599 3583 0.054611 0 - 0.36018 2024-04-22T07:24:05.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4101 16 3599 3583 0.0545977 0 - 0.36018 2024-04-22T07:24:06.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4102 16 3599 3583 0.0545844 0 - 0.36018 2024-04-22T07:24:07.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4103 16 3599 3583 0.0545711 0 - 0.36018 2024-04-22T07:24:08.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4104 16 3599 3583 0.0545578 0 - 0.36018 2024-04-22T07:24:09.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4105 16 3599 3583 0.0545445 0 - 0.36018 2024-04-22T07:24:10.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4106 16 3599 3583 0.0545312 0 - 0.36018 2024-04-22T07:24:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:24:11.028 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:24:11.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4107 16 3599 3583 0.054518 0 - 0.36018 2024-04-22T07:24:12.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4108 16 3599 3583 0.0545047 0 - 0.36018 2024-04-22T07:24:13.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:24:13.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4109 16 3599 3583 0.0544914 0 - 0.36018 2024-04-22T07:24:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:24:13.156+0000 7f4cf746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:24:14.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4110 16 3599 3583 0.0544782 0 - 0.36018 2024-04-22T07:24:15.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4111 16 3599 3583 0.0544649 0 - 0.36018 2024-04-22T07:24:16.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4112 16 3599 3583 0.0544517 0 - 0.36018 2024-04-22T07:24:17.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4113 16 3599 3583 0.0544384 0 - 0.36018 2024-04-22T07:24:18.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4114 16 3599 3583 0.0544252 0 - 0.36018 2024-04-22T07:24:19.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4115 16 3599 3583 0.054412 0 - 0.36018 2024-04-22T07:24:20.038 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4116 16 3599 3583 0.0543988 0 - 0.36018 2024-04-22T07:24:21.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4117 16 3599 3583 0.0543855 0 - 0.36018 2024-04-22T07:24:22.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4118 16 3599 3583 0.0543723 0 - 0.36018 2024-04-22T07:24:23.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4119 16 3599 3583 0.0543591 0 - 0.36018 2024-04-22T07:24:24.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:24:24.042816+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:24:24.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:24:24.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4120 16 3599 3583 0.0543459 0 - 0.36018 2024-04-22T07:24:25.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4121 16 3599 3583 0.0543328 0 - 0.36018 2024-04-22T07:24:26.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4122 16 3599 3583 0.0543196 0 - 0.36018 2024-04-22T07:24:27.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4123 16 3599 3583 0.0543064 0 - 0.36018 2024-04-22T07:24:28.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4124 16 3599 3583 0.0542932 0 - 0.36018 2024-04-22T07:24:29.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4125 16 3599 3583 0.0542801 0 - 0.36018 2024-04-22T07:24:30.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4126 16 3599 3583 0.0542669 0 - 0.36018 2024-04-22T07:24:31.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4127 16 3599 3583 0.0542538 0 - 0.36018 2024-04-22T07:24:32.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4128 16 3599 3583 0.0542406 0 - 0.36018 2024-04-22T07:24:33.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4129 16 3599 3583 0.0542275 0 - 0.36018 2024-04-22T07:24:34.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4130 16 3599 3583 0.0542144 0 - 0.36018 2024-04-22T07:24:35.041 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4131 16 3599 3583 0.0542012 0 - 0.36018 2024-04-22T07:24:36.041 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4132 16 3599 3583 0.0541881 0 - 0.36018 2024-04-22T07:24:37.041 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4133 16 3599 3583 0.054175 0 - 0.36018 2024-04-22T07:24:38.041 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4134 16 3599 3583 0.0541619 0 - 0.36018 2024-04-22T07:24:39.041 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4135 16 3599 3583 0.0541488 0 - 0.36018 2024-04-22T07:24:40.041 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4136 16 3599 3583 0.0541357 0 - 0.36018 2024-04-22T07:24:41.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4137 16 3599 3583 0.0541226 0 - 0.36018 2024-04-22T07:24:42.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4138 16 3599 3583 0.0541095 0 - 0.36018 2024-04-22T07:24:43.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4139 16 3599 3583 0.0540965 0 - 0.36018 2024-04-22T07:24:44.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:24:44.045688+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:24:44.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:24:44.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4140 16 3599 3583 0.0540834 0 - 0.36018 2024-04-22T07:24:45.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4141 16 3599 3583 0.0540703 0 - 0.36018 2024-04-22T07:24:46.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4142 16 3599 3583 0.0540573 0 - 0.36018 2024-04-22T07:24:47.042 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4143 16 3599 3583 0.0540442 0 - 0.36018 2024-04-22T07:24:48.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4144 16 3599 3583 0.0540312 0 - 0.36018 2024-04-22T07:24:49.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4145 16 3599 3583 0.0540182 0 - 0.36018 2024-04-22T07:24:50.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4146 16 3599 3583 0.0540051 0 - 0.36018 2024-04-22T07:24:51.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4147 16 3599 3583 0.0539921 0 - 0.36018 2024-04-22T07:24:52.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4148 16 3599 3583 0.0539791 0 - 0.36018 2024-04-22T07:24:53.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4149 16 3599 3583 0.0539661 0 - 0.36018 2024-04-22T07:24:54.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4150 16 3599 3583 0.0539531 0 - 0.36018 2024-04-22T07:24:55.044 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4151 16 3599 3583 0.0539401 0 - 0.36018 2024-04-22T07:24:56.044 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4152 16 3599 3583 0.0539271 0 - 0.36018 2024-04-22T07:24:57.044 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4153 16 3599 3583 0.0539141 0 - 0.36018 2024-04-22T07:24:58.044 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4154 16 3599 3583 0.0539011 0 - 0.36018 2024-04-22T07:24:59.044 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4155 16 3599 3583 0.0538882 0 - 0.36018 2024-04-22T07:25:00.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4156 16 3599 3583 0.0538752 0 - 0.36018 2024-04-22T07:25:01.044 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4157 16 3599 3583 0.0538622 0 - 0.36018 2024-04-22T07:25:02.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4158 16 3599 3583 0.0538493 0 - 0.36018 2024-04-22T07:25:03.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4159 16 3599 3583 0.0538363 0 - 0.36018 2024-04-22T07:25:04.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:25:04.048585+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:25:04.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:25:04.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4160 16 3599 3583 0.0538234 0 - 0.36018 2024-04-22T07:25:05.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4161 16 3599 3583 0.0538104 0 - 0.36018 2024-04-22T07:25:06.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4162 16 3599 3583 0.0537975 0 - 0.36018 2024-04-22T07:25:07.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4163 16 3599 3583 0.0537846 0 - 0.36018 2024-04-22T07:25:08.045 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4164 16 3599 3583 0.0537717 0 - 0.36018 2024-04-22T07:25:09.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4165 16 3599 3583 0.0537588 0 - 0.36018 2024-04-22T07:25:10.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4166 16 3599 3583 0.0537459 0 - 0.36018 2024-04-22T07:25:11.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4167 16 3599 3583 0.053733 0 - 0.36018 2024-04-22T07:25:12.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4168 16 3599 3583 0.0537201 0 - 0.36018 2024-04-22T07:25:13.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4169 16 3599 3583 0.0537072 0 - 0.36018 2024-04-22T07:25:14.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4170 16 3599 3583 0.0536943 0 - 0.36018 2024-04-22T07:25:15.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4171 16 3599 3583 0.0536814 0 - 0.36018 2024-04-22T07:25:16.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4172 16 3599 3583 0.0536686 0 - 0.36018 2024-04-22T07:25:17.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4173 16 3599 3583 0.0536557 0 - 0.36018 2024-04-22T07:25:18.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4174 16 3599 3583 0.0536429 0 - 0.36018 2024-04-22T07:25:19.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4175 16 3599 3583 0.05363 0 - 0.36018 2024-04-22T07:25:20.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4176 16 3599 3583 0.0536172 0 - 0.36018 2024-04-22T07:25:21.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4177 16 3599 3583 0.0536043 0 - 0.36018 2024-04-22T07:25:22.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4178 16 3599 3583 0.0535915 0 - 0.36018 2024-04-22T07:25:23.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4179 16 3599 3583 0.0535787 0 - 0.36018 2024-04-22T07:25:24.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:25:24.051535+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:25:24.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:25:24.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4180 16 3599 3583 0.0535659 0 - 0.36018 2024-04-22T07:25:25.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4181 16 3599 3583 0.053553 0 - 0.36018 2024-04-22T07:25:26.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4182 16 3599 3583 0.0535402 0 - 0.36018 2024-04-22T07:25:27.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4183 16 3599 3583 0.0535274 0 - 0.36018 2024-04-22T07:25:28.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4184 16 3599 3583 0.0535146 0 - 0.36018 2024-04-22T07:25:29.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4185 16 3599 3583 0.0535019 0 - 0.36018 2024-04-22T07:25:30.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4186 16 3599 3583 0.0534891 0 - 0.36018 2024-04-22T07:25:31.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4187 16 3599 3583 0.0534763 0 - 0.36018 2024-04-22T07:25:32.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4188 16 3599 3583 0.0534635 0 - 0.36018 2024-04-22T07:25:33.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4189 16 3599 3583 0.0534508 0 - 0.36018 2024-04-22T07:25:34.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4190 16 3599 3583 0.053438 0 - 0.36018 2024-04-22T07:25:35.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4191 16 3599 3583 0.0534253 0 - 0.36018 2024-04-22T07:25:36.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4192 16 3599 3583 0.0534125 0 - 0.36018 2024-04-22T07:25:37.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4193 16 3599 3583 0.0533998 0 - 0.36018 2024-04-22T07:25:38.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4194 16 3599 3583 0.053387 0 - 0.36018 2024-04-22T07:25:39.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4195 16 3599 3583 0.0533743 0 - 0.36018 2024-04-22T07:25:40.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4196 16 3599 3583 0.0533616 0 - 0.36018 2024-04-22T07:25:41.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4197 16 3599 3583 0.0533489 0 - 0.36018 2024-04-22T07:25:42.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4198 16 3599 3583 0.0533362 0 - 0.36018 2024-04-22T07:25:43.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4199 16 3599 3583 0.0533235 0 - 0.36018 2024-04-22T07:25:44.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:25:44.054556+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:25:44.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:25:44.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4200 16 3599 3583 0.0533108 0 - 0.36018 2024-04-22T07:25:45.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4201 16 3599 3583 0.0532981 0 - 0.36018 2024-04-22T07:25:46.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4202 16 3599 3583 0.0532854 0 - 0.36018 2024-04-22T07:25:47.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4203 16 3599 3583 0.0532727 0 - 0.36018 2024-04-22T07:25:48.051 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4204 16 3599 3583 0.0532601 0 - 0.36018 2024-04-22T07:25:49.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4205 16 3599 3583 0.0532474 0 - 0.36018 2024-04-22T07:25:50.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4206 16 3599 3583 0.0532347 0 - 0.36018 2024-04-22T07:25:51.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4207 16 3599 3583 0.0532221 0 - 0.36018 2024-04-22T07:25:52.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4208 16 3599 3583 0.0532094 0 - 0.36018 2024-04-22T07:25:53.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4209 16 3599 3583 0.0531968 0 - 0.36018 2024-04-22T07:25:54.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4210 16 3599 3583 0.0531841 0 - 0.36018 2024-04-22T07:25:55.052 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4211 16 3599 3583 0.0531715 0 - 0.36018 2024-04-22T07:25:56.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4212 16 3599 3583 0.0531589 0 - 0.36018 2024-04-22T07:25:57.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4213 16 3599 3583 0.0531463 0 - 0.36018 2024-04-22T07:25:58.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4214 16 3599 3583 0.0531337 0 - 0.36018 2024-04-22T07:25:59.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4215 16 3599 3583 0.0531211 0 - 0.36018 2024-04-22T07:26:00.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4216 16 3599 3583 0.0531085 0 - 0.36018 2024-04-22T07:26:01.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4217 16 3599 3583 0.0530959 0 - 0.36018 2024-04-22T07:26:02.053 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4218 16 3599 3583 0.0530833 0 - 0.36018 2024-04-22T07:26:03.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4219 16 3599 3583 0.0530707 0 - 0.36018 2024-04-22T07:26:04.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:26:04.057481+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:26:04.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:26:04.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4220 16 3599 3583 0.0530581 0 - 0.36018 2024-04-22T07:26:05.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4221 16 3599 3583 0.0530455 0 - 0.36018 2024-04-22T07:26:06.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4222 16 3599 3583 0.053033 0 - 0.36018 2024-04-22T07:26:07.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4223 16 3599 3583 0.0530204 0 - 0.36018 2024-04-22T07:26:08.054 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4224 16 3599 3583 0.0530079 0 - 0.36018 2024-04-22T07:26:09.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4225 16 3599 3583 0.0529953 0 - 0.36018 2024-04-22T07:26:10.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4226 16 3599 3583 0.0529828 0 - 0.36018 2024-04-22T07:26:11.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4227 16 3599 3583 0.0529703 0 - 0.36018 2024-04-22T07:26:12.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4228 16 3599 3583 0.0529577 0 - 0.36018 2024-04-22T07:26:13.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4229 16 3599 3583 0.0529452 0 - 0.36018 2024-04-22T07:26:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:26:13.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:26:14.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4230 16 3599 3583 0.0529327 0 - 0.36018 2024-04-22T07:26:15.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4231 16 3599 3583 0.0529202 0 - 0.36018 2024-04-22T07:26:15.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:26:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:26:15.192+0000 7faf7e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:26:16.055 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4232 16 3599 3583 0.0529077 0 - 0.36018 2024-04-22T07:26:17.056 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4233 16 3599 3583 0.0528952 0 - 0.36018 2024-04-22T07:26:18.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4234 16 3599 3583 0.0528827 0 - 0.36018 2024-04-22T07:26:19.056 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4235 16 3599 3583 0.0528702 0 - 0.36018 2024-04-22T07:26:20.056 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4236 16 3599 3583 0.0528577 0 - 0.36018 2024-04-22T07:26:21.056 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4237 16 3599 3583 0.0528452 0 - 0.36018 2024-04-22T07:26:22.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4238 16 3599 3583 0.0528328 0 - 0.36018 2024-04-22T07:26:23.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4239 16 3599 3583 0.0528203 0 - 0.36018 2024-04-22T07:26:24.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:26:24.060496+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:26:24.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:26:24.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4240 16 3599 3583 0.0528078 0 - 0.36018 2024-04-22T07:26:25.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4241 16 3599 3583 0.0527954 0 - 0.36018 2024-04-22T07:26:26.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4242 16 3599 3583 0.0527829 0 - 0.36018 2024-04-22T07:26:27.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4243 16 3599 3583 0.0527705 0 - 0.36018 2024-04-22T07:26:28.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4244 16 3599 3583 0.0527581 0 - 0.36018 2024-04-22T07:26:29.057 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4245 16 3599 3583 0.0527456 0 - 0.36018 2024-04-22T07:26:30.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4246 16 3599 3583 0.0527332 0 - 0.36018 2024-04-22T07:26:31.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4247 16 3599 3583 0.0527208 0 - 0.36018 2024-04-22T07:26:32.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4248 16 3599 3583 0.0527084 0 - 0.36018 2024-04-22T07:26:33.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4249 16 3599 3583 0.052696 0 - 0.36018 2024-04-22T07:26:34.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4250 16 3599 3583 0.0526836 0 - 0.36018 2024-04-22T07:26:35.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4251 16 3599 3583 0.0526712 0 - 0.36018 2024-04-22T07:26:36.058 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4252 16 3599 3583 0.0526588 0 - 0.36018 2024-04-22T07:26:37.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4253 16 3599 3583 0.0526464 0 - 0.36018 2024-04-22T07:26:38.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4254 16 3599 3583 0.052634 0 - 0.36018 2024-04-22T07:26:39.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4255 16 3599 3583 0.0526217 0 - 0.36018 2024-04-22T07:26:40.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4256 16 3599 3583 0.0526093 0 - 0.36018 2024-04-22T07:26:41.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4257 16 3599 3583 0.052597 0 - 0.36018 2024-04-22T07:26:42.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4258 16 3599 3583 0.0525846 0 - 0.36018 2024-04-22T07:26:43.059 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4259 16 3599 3583 0.0525723 0 - 0.36018 2024-04-22T07:26:44.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:26:44.063338+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:26:44.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:26:44.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4260 16 3599 3583 0.0525599 0 - 0.36018 2024-04-22T07:26:45.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4261 16 3599 3583 0.0525476 0 - 0.36018 2024-04-22T07:26:46.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4262 16 3599 3583 0.0525353 0 - 0.36018 2024-04-22T07:26:47.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4263 16 3599 3583 0.0525229 0 - 0.36018 2024-04-22T07:26:48.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4264 16 3599 3583 0.0525106 0 - 0.36018 2024-04-22T07:26:49.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4265 16 3599 3583 0.0524983 0 - 0.36018 2024-04-22T07:26:50.060 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4266 16 3599 3583 0.052486 0 - 0.36018 2024-04-22T07:26:51.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4267 16 3599 3583 0.0524737 0 - 0.36018 2024-04-22T07:26:52.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4268 16 3599 3583 0.0524614 0 - 0.36018 2024-04-22T07:26:53.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4269 16 3599 3583 0.0524491 0 - 0.36018 2024-04-22T07:26:54.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4270 16 3599 3583 0.0524368 0 - 0.36018 2024-04-22T07:26:55.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4271 16 3599 3583 0.0524245 0 - 0.36018 2024-04-22T07:26:56.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4272 16 3599 3583 0.0524123 0 - 0.36018 2024-04-22T07:26:57.061 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4273 16 3599 3583 0.0524 0 - 0.36018 2024-04-22T07:26:58.062 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4274 16 3599 3583 0.0523878 0 - 0.36018 2024-04-22T07:26:59.062 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4275 16 3599 3583 0.0523755 0 - 0.36018 2024-04-22T07:27:00.062 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4276 16 3599 3583 0.0523632 0 - 0.36018 2024-04-22T07:27:01.062 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4277 16 3599 3583 0.052351 0 - 0.36018 2024-04-22T07:27:02.062 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4278 16 3599 3583 0.0523388 0 - 0.36018 2024-04-22T07:27:03.062 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4279 16 3599 3583 0.0523265 0 - 0.36018 2024-04-22T07:27:04.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:27:04.066256+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:27:04.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:27:04.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4280 16 3599 3583 0.0523143 0 - 0.36018 2024-04-22T07:27:05.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4281 16 3599 3583 0.0523021 0 - 0.36018 2024-04-22T07:27:06.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4282 16 3599 3583 0.0522899 0 - 0.36018 2024-04-22T07:27:07.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4283 16 3599 3583 0.0522777 0 - 0.36018 2024-04-22T07:27:08.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4284 16 3599 3583 0.0522655 0 - 0.36018 2024-04-22T07:27:09.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4285 16 3599 3583 0.0522533 0 - 0.36018 2024-04-22T07:27:10.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4286 16 3599 3583 0.0522411 0 - 0.36018 2024-04-22T07:27:11.063 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4287 16 3599 3583 0.0522289 0 - 0.36018 2024-04-22T07:27:12.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4288 16 3599 3583 0.0522167 0 - 0.36018 2024-04-22T07:27:13.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4289 16 3599 3583 0.0522045 0 - 0.36018 2024-04-22T07:27:14.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4290 16 3599 3583 0.0521924 0 - 0.36018 2024-04-22T07:27:15.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4291 16 3599 3583 0.0521802 0 - 0.36018 2024-04-22T07:27:16.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4292 16 3599 3583 0.052168 0 - 0.36018 2024-04-22T07:27:17.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4293 16 3599 3583 0.0521559 0 - 0.36018 2024-04-22T07:27:18.064 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4294 16 3599 3583 0.0521437 0 - 0.36018 2024-04-22T07:27:19.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4295 16 3599 3583 0.0521316 0 - 0.36018 2024-04-22T07:27:20.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4296 16 3599 3583 0.0521195 0 - 0.36018 2024-04-22T07:27:21.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4297 16 3599 3583 0.0521073 0 - 0.36018 2024-04-22T07:27:22.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4298 16 3599 3583 0.0520952 0 - 0.36018 2024-04-22T07:27:23.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4299 16 3599 3583 0.0520831 0 - 0.36018 2024-04-22T07:27:24.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:27:24.069103+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:27:24.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:27:24.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4300 16 3599 3583 0.052071 0 - 0.36018 2024-04-22T07:27:25.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4301 16 3599 3583 0.0520589 0 - 0.36018 2024-04-22T07:27:26.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4302 16 3599 3583 0.0520468 0 - 0.36018 2024-04-22T07:27:27.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4303 16 3599 3583 0.0520347 0 - 0.36018 2024-04-22T07:27:28.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4304 16 3599 3583 0.0520226 0 - 0.36018 2024-04-22T07:27:29.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4305 16 3599 3583 0.0520105 0 - 0.36018 2024-04-22T07:27:30.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4306 16 3599 3583 0.0519984 0 - 0.36018 2024-04-22T07:27:31.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4307 16 3599 3583 0.0519864 0 - 0.36018 2024-04-22T07:27:32.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4308 16 3599 3583 0.0519743 0 - 0.36018 2024-04-22T07:27:33.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4309 16 3599 3583 0.0519622 0 - 0.36018 2024-04-22T07:27:34.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4310 16 3599 3583 0.0519502 0 - 0.36018 2024-04-22T07:27:35.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4311 16 3599 3583 0.0519381 0 - 0.36018 2024-04-22T07:27:36.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4312 16 3599 3583 0.0519261 0 - 0.36018 2024-04-22T07:27:37.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4313 16 3599 3583 0.051914 0 - 0.36018 2024-04-22T07:27:38.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4314 16 3599 3583 0.051902 0 - 0.36018 2024-04-22T07:27:39.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4315 16 3599 3583 0.05189 0 - 0.36018 2024-04-22T07:27:40.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4316 16 3599 3583 0.051878 0 - 0.36018 2024-04-22T07:27:41.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4317 16 3599 3583 0.0518659 0 - 0.36018 2024-04-22T07:27:42.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4318 16 3599 3583 0.0518539 0 - 0.36018 2024-04-22T07:27:43.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4319 16 3599 3583 0.0518419 0 - 0.36018 2024-04-22T07:27:44.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:27:44.072069+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:27:44.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:27:44.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4320 16 3599 3583 0.0518299 0 - 0.36018 2024-04-22T07:27:45.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4321 16 3599 3583 0.0518179 0 - 0.36018 2024-04-22T07:27:46.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4322 16 3599 3583 0.0518059 0 - 0.36018 2024-04-22T07:27:47.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4323 16 3599 3583 0.0517939 0 - 0.36018 2024-04-22T07:27:48.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4324 16 3599 3583 0.051782 0 - 0.36018 2024-04-22T07:27:49.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4325 16 3599 3583 0.05177 0 - 0.36018 2024-04-22T07:27:50.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4326 16 3599 3583 0.051758 0 - 0.36018 2024-04-22T07:27:51.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4327 16 3599 3583 0.0517461 0 - 0.36018 2024-04-22T07:27:52.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4328 16 3599 3583 0.0517341 0 - 0.36018 2024-04-22T07:27:53.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4329 16 3599 3583 0.0517222 0 - 0.36018 2024-04-22T07:27:54.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4330 16 3599 3583 0.0517102 0 - 0.36018 2024-04-22T07:27:55.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4331 16 3599 3583 0.0516983 0 - 0.36018 2024-04-22T07:27:56.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4332 16 3599 3583 0.0516863 0 - 0.36018 2024-04-22T07:27:57.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4333 16 3599 3583 0.0516744 0 - 0.36018 2024-04-22T07:27:58.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4334 16 3599 3583 0.0516625 0 - 0.36018 2024-04-22T07:27:59.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4335 16 3599 3583 0.0516506 0 - 0.36018 2024-04-22T07:28:00.071 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4336 16 3599 3583 0.0516387 0 - 0.36018 2024-04-22T07:28:01.071 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4337 16 3599 3583 0.0516268 0 - 0.36018 2024-04-22T07:28:02.071 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4338 16 3599 3583 0.0516149 0 - 0.36018 2024-04-22T07:28:03.071 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4339 16 3599 3583 0.051603 0 - 0.36018 2024-04-22T07:28:04.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:28:04.074988+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:28:04.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:28:04.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4340 16 3599 3583 0.0515911 0 - 0.36018 2024-04-22T07:28:05.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4341 16 3599 3583 0.0515792 0 - 0.36018 2024-04-22T07:28:06.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4342 16 3599 3583 0.0515673 0 - 0.36018 2024-04-22T07:28:07.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4343 16 3599 3583 0.0515554 0 - 0.36018 2024-04-22T07:28:08.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4344 16 3599 3583 0.0515436 0 - 0.36018 2024-04-22T07:28:09.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4345 16 3599 3583 0.0515317 0 - 0.36018 2024-04-22T07:28:10.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4346 16 3599 3583 0.0515198 0 - 0.36018 2024-04-22T07:28:11.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4347 16 3599 3583 0.051508 0 - 0.36018 2024-04-22T07:28:12.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4348 16 3599 3583 0.0514961 0 - 0.36018 2024-04-22T07:28:13.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4349 16 3599 3583 0.0514843 0 - 0.36018 2024-04-22T07:28:14.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4350 16 3599 3583 0.0514725 0 - 0.36018 2024-04-22T07:28:15.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4351 16 3599 3583 0.0514606 0 - 0.36018 2024-04-22T07:28:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:28:15.105 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:28:16.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4352 16 3599 3583 0.0514488 0 - 0.36018 2024-04-22T07:28:17.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4353 16 3599 3583 0.051437 0 - 0.36018 2024-04-22T07:28:17.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:28:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:28:17.231+0000 7fe31eb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:28:18.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4354 16 3599 3583 0.0514252 0 - 0.36018 2024-04-22T07:28:19.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4355 16 3599 3583 0.0514134 0 - 0.36018 2024-04-22T07:28:20.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4356 16 3599 3583 0.0514016 0 - 0.36018 2024-04-22T07:28:21.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4357 16 3599 3583 0.0513898 0 - 0.36018 2024-04-22T07:28:22.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4358 16 3599 3583 0.051378 0 - 0.36018 2024-04-22T07:28:23.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4359 16 3599 3583 0.0513662 0 - 0.36018 2024-04-22T07:28:24.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:28:24.077871+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:28:24.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:28:24.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4360 16 3599 3583 0.0513544 0 - 0.36018 2024-04-22T07:28:25.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4361 16 3599 3583 0.0513426 0 - 0.36018 2024-04-22T07:28:26.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4362 16 3599 3583 0.0513309 0 - 0.36018 2024-04-22T07:28:27.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4363 16 3599 3583 0.0513191 0 - 0.36018 2024-04-22T07:28:28.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4364 16 3599 3583 0.0513073 0 - 0.36018 2024-04-22T07:28:29.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4365 16 3599 3583 0.0512956 0 - 0.36018 2024-04-22T07:28:30.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4366 16 3599 3583 0.0512838 0 - 0.36018 2024-04-22T07:28:31.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4367 16 3599 3583 0.0512721 0 - 0.36018 2024-04-22T07:28:32.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4368 16 3599 3583 0.0512604 0 - 0.36018 2024-04-22T07:28:33.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4369 16 3599 3583 0.0512486 0 - 0.36018 2024-04-22T07:28:34.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4370 16 3599 3583 0.0512369 0 - 0.36018 2024-04-22T07:28:35.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4371 16 3599 3583 0.0512252 0 - 0.36018 2024-04-22T07:28:36.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4372 16 3599 3583 0.0512135 0 - 0.36018 2024-04-22T07:28:37.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4373 16 3599 3583 0.0512017 0 - 0.36018 2024-04-22T07:28:38.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4374 16 3599 3583 0.05119 0 - 0.36018 2024-04-22T07:28:39.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4375 16 3599 3583 0.0511783 0 - 0.36018 2024-04-22T07:28:40.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4376 16 3599 3583 0.0511666 0 - 0.36018 2024-04-22T07:28:41.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4377 16 3599 3583 0.051155 0 - 0.36018 2024-04-22T07:28:42.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4378 16 3599 3583 0.0511433 0 - 0.36018 2024-04-22T07:28:43.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4379 16 3599 3583 0.0511316 0 - 0.36018 2024-04-22T07:28:44.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:28:44.080799+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:28:44.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:28:44.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4380 16 3599 3583 0.0511199 0 - 0.36018 2024-04-22T07:28:45.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4381 16 3599 3583 0.0511082 0 - 0.36018 2024-04-22T07:28:46.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4382 16 3599 3583 0.0510966 0 - 0.36018 2024-04-22T07:28:47.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4383 16 3599 3583 0.0510849 0 - 0.36018 2024-04-22T07:28:48.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4384 16 3599 3583 0.0510733 0 - 0.36018 2024-04-22T07:28:49.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4385 16 3599 3583 0.0510616 0 - 0.36018 2024-04-22T07:28:50.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4386 16 3599 3583 0.05105 0 - 0.36018 2024-04-22T07:28:51.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4387 16 3599 3583 0.0510383 0 - 0.36018 2024-04-22T07:28:52.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4388 16 3599 3583 0.0510267 0 - 0.36018 2024-04-22T07:28:53.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4389 16 3599 3583 0.0510151 0 - 0.36018 2024-04-22T07:28:54.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4390 16 3599 3583 0.0510035 0 - 0.36018 2024-04-22T07:28:55.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4391 16 3599 3583 0.0509919 0 - 0.36018 2024-04-22T07:28:56.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4392 16 3599 3583 0.0509802 0 - 0.36018 2024-04-22T07:28:57.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4393 16 3599 3583 0.0509686 0 - 0.36018 2024-04-22T07:28:58.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4394 16 3599 3583 0.050957 0 - 0.36018 2024-04-22T07:28:59.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4395 16 3599 3583 0.0509454 0 - 0.36018 2024-04-22T07:29:00.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4396 16 3599 3583 0.0509339 0 - 0.36018 2024-04-22T07:29:01.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4397 16 3599 3583 0.0509223 0 - 0.36018 2024-04-22T07:29:02.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4398 16 3599 3583 0.0509107 0 - 0.36018 2024-04-22T07:29:03.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4399 16 3599 3583 0.0508991 0 - 0.36018 2024-04-22T07:29:04.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:29:04.083656+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:29:04.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:29:04.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4400 16 3599 3583 0.0508876 0 - 0.36018 2024-04-22T07:29:05.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4401 16 3599 3583 0.050876 0 - 0.36018 2024-04-22T07:29:06.080 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4402 16 3599 3583 0.0508644 0 - 0.36018 2024-04-22T07:29:07.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4403 16 3599 3583 0.0508529 0 - 0.36018 2024-04-22T07:29:08.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4404 16 3599 3583 0.0508413 0 - 0.36018 2024-04-22T07:29:09.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4405 16 3599 3583 0.0508298 0 - 0.36018 2024-04-22T07:29:10.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4406 16 3599 3583 0.0508183 0 - 0.36018 2024-04-22T07:29:11.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4407 16 3599 3583 0.0508067 0 - 0.36018 2024-04-22T07:29:12.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4408 16 3599 3583 0.0507952 0 - 0.36018 2024-04-22T07:29:13.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4409 16 3599 3583 0.0507837 0 - 0.36018 2024-04-22T07:29:14.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4410 16 3599 3583 0.0507722 0 - 0.36018 2024-04-22T07:29:15.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4411 16 3599 3583 0.0507607 0 - 0.36018 2024-04-22T07:29:16.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4412 16 3599 3583 0.0507491 0 - 0.36018 2024-04-22T07:29:17.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4413 16 3599 3583 0.0507376 0 - 0.36018 2024-04-22T07:29:18.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4414 16 3599 3583 0.0507262 0 - 0.36018 2024-04-22T07:29:19.082 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4415 16 3599 3583 0.0507147 0 - 0.36018 2024-04-22T07:29:20.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4416 16 3599 3583 0.0507032 0 - 0.36018 2024-04-22T07:29:21.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4417 16 3599 3583 0.0506917 0 - 0.36018 2024-04-22T07:29:22.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4418 16 3599 3583 0.0506802 0 - 0.36018 2024-04-22T07:29:23.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4419 16 3599 3583 0.0506688 0 - 0.36018 2024-04-22T07:29:24.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:29:24.086642+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:29:24.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:29:24.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4420 16 3599 3583 0.0506573 0 - 0.36018 2024-04-22T07:29:25.083 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4421 16 3599 3583 0.0506458 0 - 0.36018 2024-04-22T07:29:26.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4422 16 3599 3583 0.0506344 0 - 0.36018 2024-04-22T07:29:27.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4423 16 3599 3583 0.0506229 0 - 0.36018 2024-04-22T07:29:28.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4424 16 3599 3583 0.0506115 0 - 0.36018 2024-04-22T07:29:29.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4425 16 3599 3583 0.0506001 0 - 0.36018 2024-04-22T07:29:30.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4426 16 3599 3583 0.0505886 0 - 0.36018 2024-04-22T07:29:31.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4427 16 3599 3583 0.0505772 0 - 0.36018 2024-04-22T07:29:32.084 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4428 16 3599 3583 0.0505658 0 - 0.36018 2024-04-22T07:29:33.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4429 16 3599 3583 0.0505544 0 - 0.36018 2024-04-22T07:29:34.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4430 16 3599 3583 0.0505429 0 - 0.36018 2024-04-22T07:29:35.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4431 16 3599 3583 0.0505315 0 - 0.36018 2024-04-22T07:29:36.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4432 16 3599 3583 0.0505201 0 - 0.36018 2024-04-22T07:29:37.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4433 16 3599 3583 0.0505087 0 - 0.36018 2024-04-22T07:29:38.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4434 16 3599 3583 0.0504973 0 - 0.36018 2024-04-22T07:29:39.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4435 16 3599 3583 0.050486 0 - 0.36018 2024-04-22T07:29:40.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4436 16 3599 3583 0.0504746 0 - 0.36018 2024-04-22T07:29:41.085 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4437 16 3599 3583 0.0504632 0 - 0.36018 2024-04-22T07:29:42.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4438 16 3599 3583 0.0504518 0 - 0.36018 2024-04-22T07:29:43.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4439 16 3599 3583 0.0504405 0 - 0.36018 2024-04-22T07:29:44.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:29:44.089430+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:29:44.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:29:44.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4440 16 3599 3583 0.0504291 0 - 0.36018 2024-04-22T07:29:45.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4441 16 3599 3583 0.0504178 0 - 0.36018 2024-04-22T07:29:46.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4442 16 3599 3583 0.0504064 0 - 0.36018 2024-04-22T07:29:47.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4443 16 3599 3583 0.0503951 0 - 0.36018 2024-04-22T07:29:48.086 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4444 16 3599 3583 0.0503837 0 - 0.36018 2024-04-22T07:29:49.087 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4445 16 3599 3583 0.0503724 0 - 0.36018 2024-04-22T07:29:50.087 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4446 16 3599 3583 0.0503611 0 - 0.36018 2024-04-22T07:29:51.087 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4447 16 3599 3583 0.0503497 0 - 0.36018 2024-04-22T07:29:52.087 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4448 16 3599 3583 0.0503384 0 - 0.36018 2024-04-22T07:29:53.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4449 16 3599 3583 0.0503271 0 - 0.36018 2024-04-22T07:29:54.088 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4450 16 3599 3583 0.0503158 0 - 0.36018 2024-04-22T07:29:55.088 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4451 16 3599 3583 0.0503045 0 - 0.36018 2024-04-22T07:29:56.088 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4452 16 3599 3583 0.0502932 0 - 0.36018 2024-04-22T07:29:57.088 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4453 16 3599 3583 0.0502819 0 - 0.36018 2024-04-22T07:29:58.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4454 16 3599 3583 0.0502706 0 - 0.36018 2024-04-22T07:29:59.088 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4455 16 3599 3583 0.0502593 0 - 0.36018 2024-04-22T07:30:00.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4456 16 3599 3583 0.050248 0 - 0.36018 2024-04-22T07:30:01.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4457 16 3599 3583 0.0502368 0 - 0.36018 2024-04-22T07:30:02.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4458 16 3599 3583 0.0502255 0 - 0.36018 2024-04-22T07:30:03.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4459 16 3599 3583 0.0502142 0 - 0.36018 2024-04-22T07:30:04.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:30:04.092435+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:30:04.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:30:04.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4460 16 3599 3583 0.050203 0 - 0.36018 2024-04-22T07:30:05.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4461 16 3599 3583 0.0501917 0 - 0.36018 2024-04-22T07:30:06.089 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4462 16 3599 3583 0.0501805 0 - 0.36018 2024-04-22T07:30:07.090 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4463 16 3599 3583 0.0501692 0 - 0.36018 2024-04-22T07:30:08.090 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4464 16 3599 3583 0.050158 0 - 0.36018 2024-04-22T07:30:09.090 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4465 16 3599 3583 0.0501467 0 - 0.36018 2024-04-22T07:30:10.090 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4466 16 3599 3583 0.0501355 0 - 0.36018 2024-04-22T07:30:11.090 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4467 16 3599 3583 0.0501243 0 - 0.36018 2024-04-22T07:30:12.090 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4468 16 3599 3583 0.0501131 0 - 0.36018 2024-04-22T07:30:13.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4469 16 3599 3583 0.0501019 0 - 0.36018 2024-04-22T07:30:14.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4470 16 3599 3583 0.0500907 0 - 0.36018 2024-04-22T07:30:15.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4471 16 3599 3583 0.0500795 0 - 0.36018 2024-04-22T07:30:16.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4472 16 3599 3583 0.0500683 0 - 0.36018 2024-04-22T07:30:17.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4473 16 3599 3583 0.0500571 0 - 0.36018 2024-04-22T07:30:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:30:17.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:30:18.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4474 16 3599 3583 0.0500459 0 - 0.36018 2024-04-22T07:30:19.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4475 16 3599 3583 0.0500347 0 - 0.36018 2024-04-22T07:30:19.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:30:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:30:19.269+0000 7f9161d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:30:20.091 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4476 16 3599 3583 0.0500235 0 - 0.36018 2024-04-22T07:30:21.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4477 16 3599 3583 0.0500123 0 - 0.36018 2024-04-22T07:30:22.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4478 16 3599 3583 0.0500012 0 - 0.36018 2024-04-22T07:30:23.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4479 16 3599 3583 0.04999 0 - 0.36018 2024-04-22T07:30:24.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:30:24.095391+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:30:24.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:30:24.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4480 16 3599 3583 0.0499788 0 - 0.36018 2024-04-22T07:30:25.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4481 16 3599 3583 0.0499677 0 - 0.36018 2024-04-22T07:30:26.092 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4482 16 3599 3583 0.0499565 0 - 0.36018 2024-04-22T07:30:27.093 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4483 16 3599 3583 0.0499454 0 - 0.36018 2024-04-22T07:30:28.093 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4484 16 3599 3583 0.0499343 0 - 0.36018 2024-04-22T07:30:29.093 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4485 16 3599 3583 0.0499231 0 - 0.36018 2024-04-22T07:30:30.093 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4486 16 3599 3583 0.049912 0 - 0.36018 2024-04-22T07:30:31.093 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4487 16 3599 3583 0.0499009 0 - 0.36018 2024-04-22T07:30:32.094 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4488 16 3599 3583 0.0498898 0 - 0.36018 2024-04-22T07:30:33.093 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4489 16 3599 3583 0.0498786 0 - 0.36018 2024-04-22T07:30:34.094 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4490 16 3599 3583 0.0498675 0 - 0.36018 2024-04-22T07:30:35.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4491 16 3599 3583 0.0498564 0 - 0.36018 2024-04-22T07:30:36.094 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4492 16 3599 3583 0.0498453 0 - 0.36018 2024-04-22T07:30:37.094 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4493 16 3599 3583 0.0498342 0 - 0.36018 2024-04-22T07:30:38.094 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4494 16 3599 3583 0.0498231 0 - 0.36018 2024-04-22T07:30:39.094 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4495 16 3599 3583 0.0498121 0 - 0.36018 2024-04-22T07:30:40.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4496 16 3599 3583 0.049801 0 - 0.36018 2024-04-22T07:30:41.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4497 16 3599 3583 0.0497899 0 - 0.36018 2024-04-22T07:30:42.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4498 16 3599 3583 0.0497788 0 - 0.36018 2024-04-22T07:30:43.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4499 16 3599 3583 0.0497678 0 - 0.36018 2024-04-22T07:30:44.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:30:44.098397+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:30:44.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:30:44.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4500 16 3599 3583 0.0497567 0 - 0.36018 2024-04-22T07:30:45.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4501 16 3599 3583 0.0497457 0 - 0.36018 2024-04-22T07:30:46.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4502 16 3599 3583 0.0497346 0 - 0.36018 2024-04-22T07:30:47.095 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4503 16 3599 3583 0.0497236 0 - 0.36018 2024-04-22T07:30:48.096 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4504 16 3599 3583 0.0497125 0 - 0.36018 2024-04-22T07:30:49.096 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4505 16 3599 3583 0.0497015 0 - 0.36018 2024-04-22T07:30:50.096 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4506 16 3599 3583 0.0496905 0 - 0.36018 2024-04-22T07:30:51.096 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4507 16 3599 3583 0.0496794 0 - 0.36018 2024-04-22T07:30:52.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4508 16 3599 3583 0.0496684 0 - 0.36018 2024-04-22T07:30:53.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4509 16 3599 3583 0.0496574 0 - 0.36018 2024-04-22T07:30:54.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4510 16 3599 3583 0.0496464 0 - 0.36018 2024-04-22T07:30:55.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4511 16 3599 3583 0.0496354 0 - 0.36018 2024-04-22T07:30:56.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4512 16 3599 3583 0.0496244 0 - 0.36018 2024-04-22T07:30:57.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4513 16 3599 3583 0.0496134 0 - 0.36018 2024-04-22T07:30:58.097 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4514 16 3599 3583 0.0496024 0 - 0.36018 2024-04-22T07:30:59.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4515 16 3599 3583 0.0495914 0 - 0.36018 2024-04-22T07:31:00.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4516 16 3599 3583 0.0495804 0 - 0.36018 2024-04-22T07:31:01.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4517 16 3599 3583 0.0495695 0 - 0.36018 2024-04-22T07:31:02.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4518 16 3599 3583 0.0495585 0 - 0.36018 2024-04-22T07:31:03.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4519 16 3599 3583 0.0495475 0 - 0.36018 2024-04-22T07:31:04.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:31:04.101354+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:31:04.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:31:04.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4520 16 3599 3583 0.0495366 0 - 0.36018 2024-04-22T07:31:05.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4521 16 3599 3583 0.0495256 0 - 0.36018 2024-04-22T07:31:06.098 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4522 16 3599 3583 0.0495146 0 - 0.36018 2024-04-22T07:31:07.099 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4523 16 3599 3583 0.0495037 0 - 0.36018 2024-04-22T07:31:08.099 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4524 16 3599 3583 0.0494928 0 - 0.36018 2024-04-22T07:31:09.099 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4525 16 3599 3583 0.0494818 0 - 0.36018 2024-04-22T07:31:10.099 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4526 16 3599 3583 0.0494709 0 - 0.36018 2024-04-22T07:31:11.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4527 16 3599 3583 0.04946 0 - 0.36018 2024-04-22T07:31:12.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4528 16 3599 3583 0.049449 0 - 0.36018 2024-04-22T07:31:13.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4529 16 3599 3583 0.0494381 0 - 0.36018 2024-04-22T07:31:14.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4530 16 3599 3583 0.0494272 0 - 0.36018 2024-04-22T07:31:15.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4531 16 3599 3583 0.0494163 0 - 0.36018 2024-04-22T07:31:16.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4532 16 3599 3583 0.0494054 0 - 0.36018 2024-04-22T07:31:17.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4533 16 3599 3583 0.0493945 0 - 0.36018 2024-04-22T07:31:18.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4534 16 3599 3583 0.0493836 0 - 0.36018 2024-04-22T07:31:19.100 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4535 16 3599 3583 0.0493727 0 - 0.36018 2024-04-22T07:31:20.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4536 16 3599 3583 0.0493618 0 - 0.36018 2024-04-22T07:31:21.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4537 16 3599 3583 0.0493509 0 - 0.36018 2024-04-22T07:31:22.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4538 16 3599 3583 0.0493401 0 - 0.36018 2024-04-22T07:31:23.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4539 16 3599 3583 0.0493292 0 - 0.36018 2024-04-22T07:31:24.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:31:24.104290+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:31:24.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:31:24.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4540 16 3599 3583 0.0493183 0 - 0.36018 2024-04-22T07:31:25.101 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4541 16 3599 3583 0.0493075 0 - 0.36018 2024-04-22T07:31:26.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4542 16 3599 3583 0.0492966 0 - 0.36018 2024-04-22T07:31:27.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4543 16 3599 3583 0.0492858 0 - 0.36018 2024-04-22T07:31:28.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4544 16 3599 3583 0.0492749 0 - 0.36018 2024-04-22T07:31:29.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4545 16 3599 3583 0.0492641 0 - 0.36018 2024-04-22T07:31:30.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4546 16 3599 3583 0.0492532 0 - 0.36018 2024-04-22T07:31:31.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4547 16 3599 3583 0.0492424 0 - 0.36018 2024-04-22T07:31:32.102 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4548 16 3599 3583 0.0492316 0 - 0.36018 2024-04-22T07:31:33.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4549 16 3599 3583 0.0492208 0 - 0.36018 2024-04-22T07:31:34.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4550 16 3599 3583 0.0492099 0 - 0.36018 2024-04-22T07:31:35.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4551 16 3599 3583 0.0491991 0 - 0.36018 2024-04-22T07:31:36.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4552 16 3599 3583 0.0491883 0 - 0.36018 2024-04-22T07:31:37.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4553 16 3599 3583 0.0491775 0 - 0.36018 2024-04-22T07:31:38.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4554 16 3599 3583 0.0491667 0 - 0.36018 2024-04-22T07:31:39.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4555 16 3599 3583 0.0491559 0 - 0.36018 2024-04-22T07:31:40.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4556 16 3599 3583 0.0491451 0 - 0.36018 2024-04-22T07:31:41.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4557 16 3599 3583 0.0491343 0 - 0.36018 2024-04-22T07:31:42.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4558 16 3599 3583 0.0491236 0 - 0.36018 2024-04-22T07:31:43.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4559 16 3599 3583 0.0491128 0 - 0.36018 2024-04-22T07:31:44.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:31:44.107303+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:31:44.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:31:44.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4560 16 3599 3583 0.049102 0 - 0.36018 2024-04-22T07:31:45.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4561 16 3599 3583 0.0490913 0 - 0.36018 2024-04-22T07:31:46.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4562 16 3599 3583 0.0490805 0 - 0.36018 2024-04-22T07:31:47.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4563 16 3599 3583 0.0490697 0 - 0.36018 2024-04-22T07:31:48.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4564 16 3599 3583 0.049059 0 - 0.36018 2024-04-22T07:31:49.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4565 16 3599 3583 0.0490482 0 - 0.36018 2024-04-22T07:31:50.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4566 16 3599 3583 0.0490375 0 - 0.36018 2024-04-22T07:31:51.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4567 16 3599 3583 0.0490268 0 - 0.36018 2024-04-22T07:31:52.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4568 16 3599 3583 0.049016 0 - 0.36018 2024-04-22T07:31:53.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4569 16 3599 3583 0.0490053 0 - 0.36018 2024-04-22T07:31:54.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4570 16 3599 3583 0.0489946 0 - 0.36018 2024-04-22T07:31:55.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4571 16 3599 3583 0.0489839 0 - 0.36018 2024-04-22T07:31:56.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4572 16 3599 3583 0.0489731 0 - 0.36018 2024-04-22T07:31:57.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4573 16 3599 3583 0.0489624 0 - 0.36018 2024-04-22T07:31:58.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4574 16 3599 3583 0.0489517 0 - 0.36018 2024-04-22T07:31:59.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4575 16 3599 3583 0.048941 0 - 0.36018 2024-04-22T07:32:00.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4576 16 3599 3583 0.0489303 0 - 0.36018 2024-04-22T07:32:01.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4577 16 3599 3583 0.0489196 0 - 0.36018 2024-04-22T07:32:02.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4578 16 3599 3583 0.048909 0 - 0.36018 2024-04-22T07:32:03.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4579 16 3599 3583 0.0488983 0 - 0.36018 2024-04-22T07:32:04.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:32:04.110038+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:32:04.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:32:04.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4580 16 3599 3583 0.0488876 0 - 0.36018 2024-04-22T07:32:05.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4581 16 3599 3583 0.0488769 0 - 0.36018 2024-04-22T07:32:06.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4582 16 3599 3583 0.0488663 0 - 0.36018 2024-04-22T07:32:07.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4583 16 3599 3583 0.0488556 0 - 0.36018 2024-04-22T07:32:08.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4584 16 3599 3583 0.0488449 0 - 0.36018 2024-04-22T07:32:09.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4585 16 3599 3583 0.0488343 0 - 0.36018 2024-04-22T07:32:10.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4586 16 3599 3583 0.0488236 0 - 0.36018 2024-04-22T07:32:11.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4587 16 3599 3583 0.048813 0 - 0.36018 2024-04-22T07:32:12.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4588 16 3599 3583 0.0488024 0 - 0.36018 2024-04-22T07:32:13.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4589 16 3599 3583 0.0487917 0 - 0.36018 2024-04-22T07:32:14.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4590 16 3599 3583 0.0487811 0 - 0.36018 2024-04-22T07:32:15.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4591 16 3599 3583 0.0487705 0 - 0.36018 2024-04-22T07:32:16.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4592 16 3599 3583 0.0487598 0 - 0.36018 2024-04-22T07:32:17.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4593 16 3599 3583 0.0487492 0 - 0.36018 2024-04-22T07:32:18.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4594 16 3599 3583 0.0487386 0 - 0.36018 2024-04-22T07:32:19.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4595 16 3599 3583 0.048728 0 - 0.36018 2024-04-22T07:32:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:32:19.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:32:20.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4596 16 3599 3583 0.0487174 0 - 0.36018 2024-04-22T07:32:21.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4597 16 3599 3583 0.0487068 0 - 0.36018 2024-04-22T07:32:21.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:32:21.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:32:21.306+0000 7efed915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:32:22.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4598 16 3599 3583 0.0486962 0 - 0.36018 2024-04-22T07:32:23.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4599 16 3599 3583 0.0486856 0 - 0.36018 2024-04-22T07:32:24.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:32:24.112811+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:32:24.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:32:24.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4600 16 3599 3583 0.048675 0 - 0.36018 2024-04-22T07:32:25.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4601 16 3599 3583 0.0486645 0 - 0.36018 2024-04-22T07:32:26.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4602 16 3599 3583 0.0486539 0 - 0.36018 2024-04-22T07:32:27.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4603 16 3599 3583 0.0486433 0 - 0.36018 2024-04-22T07:32:28.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4604 16 3599 3583 0.0486328 0 - 0.36018 2024-04-22T07:32:29.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4605 16 3599 3583 0.0486222 0 - 0.36018 2024-04-22T07:32:30.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4606 16 3599 3583 0.0486116 0 - 0.36018 2024-04-22T07:32:31.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4607 16 3599 3583 0.0486011 0 - 0.36018 2024-04-22T07:32:32.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4608 16 3599 3583 0.0485905 0 - 0.36018 2024-04-22T07:32:33.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4609 16 3599 3583 0.04858 0 - 0.36018 2024-04-22T07:32:34.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4610 16 3599 3583 0.0485695 0 - 0.36018 2024-04-22T07:32:35.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4611 16 3599 3583 0.0485589 0 - 0.36018 2024-04-22T07:32:36.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4612 16 3599 3583 0.0485484 0 - 0.36018 2024-04-22T07:32:37.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4613 16 3599 3583 0.0485379 0 - 0.36018 2024-04-22T07:32:38.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4614 16 3599 3583 0.0485274 0 - 0.36018 2024-04-22T07:32:39.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4615 16 3599 3583 0.0485168 0 - 0.36018 2024-04-22T07:32:40.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4616 16 3599 3583 0.0485063 0 - 0.36018 2024-04-22T07:32:41.112 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4617 16 3599 3583 0.0484958 0 - 0.36018 2024-04-22T07:32:42.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4618 16 3599 3583 0.0484853 0 - 0.36018 2024-04-22T07:32:43.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4619 16 3599 3583 0.0484748 0 - 0.36018 2024-04-22T07:32:44.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:32:44.115732+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:32:44.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:32:44.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4620 16 3599 3583 0.0484643 0 - 0.36018 2024-04-22T07:32:45.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4621 16 3599 3583 0.0484538 0 - 0.36018 2024-04-22T07:32:46.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4622 16 3599 3583 0.0484434 0 - 0.36018 2024-04-22T07:32:47.113 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4623 16 3599 3583 0.0484329 0 - 0.36018 2024-04-22T07:32:48.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4624 16 3599 3583 0.0484224 0 - 0.36018 2024-04-22T07:32:49.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4625 16 3599 3583 0.0484119 0 - 0.36018 2024-04-22T07:32:50.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4626 16 3599 3583 0.0484015 0 - 0.36018 2024-04-22T07:32:51.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4627 16 3599 3583 0.048391 0 - 0.36018 2024-04-22T07:32:52.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4628 16 3599 3583 0.0483806 0 - 0.36018 2024-04-22T07:32:53.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4629 16 3599 3583 0.0483701 0 - 0.36018 2024-04-22T07:32:54.114 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4630 16 3599 3583 0.0483597 0 - 0.36018 2024-04-22T07:32:55.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4631 16 3599 3583 0.0483492 0 - 0.36018 2024-04-22T07:32:56.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4632 16 3599 3583 0.0483388 0 - 0.36018 2024-04-22T07:32:57.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4633 16 3599 3583 0.0483283 0 - 0.36018 2024-04-22T07:32:58.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4634 16 3599 3583 0.0483179 0 - 0.36018 2024-04-22T07:32:59.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4635 16 3599 3583 0.0483075 0 - 0.36018 2024-04-22T07:33:00.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4636 16 3599 3583 0.0482971 0 - 0.36018 2024-04-22T07:33:01.115 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4637 16 3599 3583 0.0482867 0 - 0.36018 2024-04-22T07:33:02.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4638 16 3599 3583 0.0482762 0 - 0.36018 2024-04-22T07:33:03.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4639 16 3599 3583 0.0482658 0 - 0.36018 2024-04-22T07:33:04.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:33:04.118761+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:33:04.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:33:04.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4640 16 3599 3583 0.0482554 0 - 0.36018 2024-04-22T07:33:05.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4641 16 3599 3583 0.048245 0 - 0.36018 2024-04-22T07:33:06.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4642 16 3599 3583 0.0482346 0 - 0.36018 2024-04-22T07:33:07.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4643 16 3599 3583 0.0482243 0 - 0.36018 2024-04-22T07:33:08.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4644 16 3599 3583 0.0482139 0 - 0.36018 2024-04-22T07:33:09.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4645 16 3599 3583 0.0482035 0 - 0.36018 2024-04-22T07:33:10.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4646 16 3599 3583 0.0481931 0 - 0.36018 2024-04-22T07:33:11.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4647 16 3599 3583 0.0481827 0 - 0.36018 2024-04-22T07:33:12.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4648 16 3599 3583 0.0481724 0 - 0.36018 2024-04-22T07:33:13.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4649 16 3599 3583 0.048162 0 - 0.36018 2024-04-22T07:33:14.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4650 16 3599 3583 0.0481517 0 - 0.36018 2024-04-22T07:33:15.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4651 16 3599 3583 0.0481413 0 - 0.36018 2024-04-22T07:33:16.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4652 16 3599 3583 0.048131 0 - 0.36018 2024-04-22T07:33:17.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4653 16 3599 3583 0.0481206 0 - 0.36018 2024-04-22T07:33:18.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4654 16 3599 3583 0.0481103 0 - 0.36018 2024-04-22T07:33:19.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4655 16 3599 3583 0.0480999 0 - 0.36018 2024-04-22T07:33:20.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4656 16 3599 3583 0.0480896 0 - 0.36018 2024-04-22T07:33:21.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4657 16 3599 3583 0.0480793 0 - 0.36018 2024-04-22T07:33:22.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4658 16 3599 3583 0.048069 0 - 0.36018 2024-04-22T07:33:23.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4659 16 3599 3583 0.0480586 0 - 0.36018 2024-04-22T07:33:24.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:33:24.121710+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:33:24.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:33:24.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4660 16 3599 3583 0.0480483 0 - 0.36018 2024-04-22T07:33:25.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4661 16 3599 3583 0.048038 0 - 0.36018 2024-04-22T07:33:26.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4662 16 3599 3583 0.0480277 0 - 0.36018 2024-04-22T07:33:27.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4663 16 3599 3583 0.0480174 0 - 0.36018 2024-04-22T07:33:28.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4664 16 3599 3583 0.0480071 0 - 0.36018 2024-04-22T07:33:29.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4665 16 3599 3583 0.0479968 0 - 0.36018 2024-04-22T07:33:30.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4666 16 3599 3583 0.0479865 0 - 0.36018 2024-04-22T07:33:31.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4667 16 3599 3583 0.0479763 0 - 0.36018 2024-04-22T07:33:32.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4668 16 3599 3583 0.047966 0 - 0.36018 2024-04-22T07:33:33.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4669 16 3599 3583 0.0479557 0 - 0.36018 2024-04-22T07:33:34.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4670 16 3599 3583 0.0479454 0 - 0.36018 2024-04-22T07:33:35.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4671 16 3599 3583 0.0479352 0 - 0.36018 2024-04-22T07:33:36.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4672 16 3599 3583 0.0479249 0 - 0.36018 2024-04-22T07:33:37.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4673 16 3599 3583 0.0479147 0 - 0.36018 2024-04-22T07:33:38.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4674 16 3599 3583 0.0479044 0 - 0.36018 2024-04-22T07:33:39.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4675 16 3599 3583 0.0478942 0 - 0.36018 2024-04-22T07:33:40.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4676 16 3599 3583 0.0478839 0 - 0.36018 2024-04-22T07:33:41.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4677 16 3599 3583 0.0478737 0 - 0.36018 2024-04-22T07:33:42.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4678 16 3599 3583 0.0478634 0 - 0.36018 2024-04-22T07:33:43.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4679 16 3599 3583 0.0478532 0 - 0.36018 2024-04-22T07:33:44.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:33:44.124442+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:33:44.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:33:44.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4680 16 3599 3583 0.047843 0 - 0.36018 2024-04-22T07:33:45.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4681 16 3599 3583 0.0478328 0 - 0.36018 2024-04-22T07:33:46.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4682 16 3599 3583 0.0478226 0 - 0.36018 2024-04-22T07:33:47.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4683 16 3599 3583 0.0478123 0 - 0.36018 2024-04-22T07:33:48.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4684 16 3599 3583 0.0478021 0 - 0.36018 2024-04-22T07:33:49.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4685 16 3599 3583 0.0477919 0 - 0.36018 2024-04-22T07:33:50.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4686 16 3599 3583 0.0477817 0 - 0.36018 2024-04-22T07:33:51.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4687 16 3599 3583 0.0477715 0 - 0.36018 2024-04-22T07:33:52.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4688 16 3599 3583 0.0477614 0 - 0.36018 2024-04-22T07:33:53.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4689 16 3599 3583 0.0477512 0 - 0.36018 2024-04-22T07:33:54.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4690 16 3599 3583 0.047741 0 - 0.36018 2024-04-22T07:33:55.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4691 16 3599 3583 0.0477308 0 - 0.36018 2024-04-22T07:33:56.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4692 16 3599 3583 0.0477206 0 - 0.36018 2024-04-22T07:33:57.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4693 16 3599 3583 0.0477105 0 - 0.36018 2024-04-22T07:33:58.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4694 16 3599 3583 0.0477003 0 - 0.36018 2024-04-22T07:33:59.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4695 16 3599 3583 0.0476901 0 - 0.36018 2024-04-22T07:34:00.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4696 16 3599 3583 0.04768 0 - 0.36018 2024-04-22T07:34:01.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4697 16 3599 3583 0.0476698 0 - 0.36018 2024-04-22T07:34:02.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4698 16 3599 3583 0.0476597 0 - 0.36018 2024-04-22T07:34:03.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4699 16 3599 3583 0.0476495 0 - 0.36018 2024-04-22T07:34:04.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:34:04.127326+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:34:04.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:34:04.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4700 16 3599 3583 0.0476394 0 - 0.36018 2024-04-22T07:34:05.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4701 16 3599 3583 0.0476293 0 - 0.36018 2024-04-22T07:34:06.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4702 16 3599 3583 0.0476191 0 - 0.36018 2024-04-22T07:34:07.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4703 16 3599 3583 0.047609 0 - 0.36018 2024-04-22T07:34:08.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4704 16 3599 3583 0.0475989 0 - 0.36018 2024-04-22T07:34:09.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4705 16 3599 3583 0.0475888 0 - 0.36018 2024-04-22T07:34:10.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4706 16 3599 3583 0.0475787 0 - 0.36018 2024-04-22T07:34:11.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4707 16 3599 3583 0.0475686 0 - 0.36018 2024-04-22T07:34:12.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4708 16 3599 3583 0.0475585 0 - 0.36018 2024-04-22T07:34:13.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4709 16 3599 3583 0.0475484 0 - 0.36018 2024-04-22T07:34:14.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4710 16 3599 3583 0.0475383 0 - 0.36018 2024-04-22T07:34:15.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4711 16 3599 3583 0.0475282 0 - 0.36018 2024-04-22T07:34:16.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4712 16 3599 3583 0.0475181 0 - 0.36018 2024-04-22T07:34:17.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4713 16 3599 3583 0.047508 0 - 0.36018 2024-04-22T07:34:18.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4714 16 3599 3583 0.0474979 0 - 0.36018 2024-04-22T07:34:19.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4715 16 3599 3583 0.0474879 0 - 0.36018 2024-04-22T07:34:20.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4716 16 3599 3583 0.0474778 0 - 0.36018 2024-04-22T07:34:21.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4717 16 3599 3583 0.0474677 0 - 0.36018 2024-04-22T07:34:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:34:21.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:34:22.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4718 16 3599 3583 0.0474577 0 - 0.36018 2024-04-22T07:34:23.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4719 16 3599 3583 0.0474476 0 - 0.36018 2024-04-22T07:34:23.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T07:34:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:34:23.343+0000 7f6ce022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:34:24.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:34:24.130246+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:34:24.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:34:24.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4720 16 3599 3583 0.0474375 0 - 0.36018 2024-04-22T07:34:25.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4721 16 3599 3583 0.0474275 0 - 0.36018 2024-04-22T07:34:26.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4722 16 3599 3583 0.0474175 0 - 0.36018 2024-04-22T07:34:27.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4723 16 3599 3583 0.0474074 0 - 0.36018 2024-04-22T07:34:28.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4724 16 3599 3583 0.0473974 0 - 0.36018 2024-04-22T07:34:29.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4725 16 3599 3583 0.0473873 0 - 0.36018 2024-04-22T07:34:30.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4726 16 3599 3583 0.0473773 0 - 0.36018 2024-04-22T07:34:31.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4727 16 3599 3583 0.0473673 0 - 0.36018 2024-04-22T07:34:32.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4728 16 3599 3583 0.0473573 0 - 0.36018 2024-04-22T07:34:33.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4729 16 3599 3583 0.0473473 0 - 0.36018 2024-04-22T07:34:34.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4730 16 3599 3583 0.0473373 0 - 0.36018 2024-04-22T07:34:35.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4731 16 3599 3583 0.0473272 0 - 0.36018 2024-04-22T07:34:36.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4732 16 3599 3583 0.0473172 0 - 0.36018 2024-04-22T07:34:37.129 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4733 16 3599 3583 0.0473072 0 - 0.36018 2024-04-22T07:34:38.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4734 16 3599 3583 0.0472973 0 - 0.36018 2024-04-22T07:34:39.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4735 16 3599 3583 0.0472873 0 - 0.36018 2024-04-22T07:34:40.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4736 16 3599 3583 0.0472773 0 - 0.36018 2024-04-22T07:34:41.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4737 16 3599 3583 0.0472673 0 - 0.36018 2024-04-22T07:34:42.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4738 16 3599 3583 0.0472573 0 - 0.36018 2024-04-22T07:34:43.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4739 16 3599 3583 0.0472474 0 - 0.36018 2024-04-22T07:34:44.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:34:44.133244+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:34:44.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:34:44.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4740 16 3599 3583 0.0472374 0 - 0.36018 2024-04-22T07:34:45.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4741 16 3599 3583 0.0472274 0 - 0.36018 2024-04-22T07:34:46.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4742 16 3599 3583 0.0472175 0 - 0.36018 2024-04-22T07:34:47.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4743 16 3599 3583 0.0472075 0 - 0.36018 2024-04-22T07:34:48.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4744 16 3599 3583 0.0471976 0 - 0.36018 2024-04-22T07:34:49.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4745 16 3599 3583 0.0471876 0 - 0.36018 2024-04-22T07:34:50.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4746 16 3599 3583 0.0471777 0 - 0.36018 2024-04-22T07:34:51.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4747 16 3599 3583 0.0471677 0 - 0.36018 2024-04-22T07:34:52.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4748 16 3599 3583 0.0471578 0 - 0.36018 2024-04-22T07:34:53.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4749 16 3599 3583 0.0471479 0 - 0.36018 2024-04-22T07:34:54.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4750 16 3599 3583 0.0471379 0 - 0.36018 2024-04-22T07:34:55.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4751 16 3599 3583 0.047128 0 - 0.36018 2024-04-22T07:34:56.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4752 16 3599 3583 0.0471181 0 - 0.36018 2024-04-22T07:34:57.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4753 16 3599 3583 0.0471082 0 - 0.36018 2024-04-22T07:34:58.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4754 16 3599 3583 0.0470983 0 - 0.36018 2024-04-22T07:34:59.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4755 16 3599 3583 0.0470884 0 - 0.36018 2024-04-22T07:35:00.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4756 16 3599 3583 0.0470785 0 - 0.36018 2024-04-22T07:35:01.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4757 16 3599 3583 0.0470686 0 - 0.36018 2024-04-22T07:35:02.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4758 16 3599 3583 0.0470587 0 - 0.36018 2024-04-22T07:35:03.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4759 16 3599 3583 0.0470488 0 - 0.36018 2024-04-22T07:35:04.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:35:04.136145+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:35:04.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:35:04.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4760 16 3599 3583 0.0470389 0 - 0.36018 2024-04-22T07:35:05.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4761 16 3599 3583 0.047029 0 - 0.36018 2024-04-22T07:35:06.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4762 16 3599 3583 0.0470192 0 - 0.36018 2024-04-22T07:35:07.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4763 16 3599 3583 0.0470093 0 - 0.36018 2024-04-22T07:35:08.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4764 16 3599 3583 0.0469994 0 - 0.36018 2024-04-22T07:35:09.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4765 16 3599 3583 0.0469896 0 - 0.36018 2024-04-22T07:35:10.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4766 16 3599 3583 0.0469797 0 - 0.36018 2024-04-22T07:35:11.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4767 16 3599 3583 0.0469698 0 - 0.36018 2024-04-22T07:35:12.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4768 16 3599 3583 0.04696 0 - 0.36018 2024-04-22T07:35:13.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4769 16 3599 3583 0.0469501 0 - 0.36018 2024-04-22T07:35:14.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4770 16 3599 3583 0.0469403 0 - 0.36018 2024-04-22T07:35:15.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4771 16 3599 3583 0.0469305 0 - 0.36018 2024-04-22T07:35:16.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4772 16 3599 3583 0.0469206 0 - 0.36018 2024-04-22T07:35:17.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4773 16 3599 3583 0.0469108 0 - 0.36018 2024-04-22T07:35:18.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4774 16 3599 3583 0.046901 0 - 0.36018 2024-04-22T07:35:19.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4775 16 3599 3583 0.0468911 0 - 0.36018 2024-04-22T07:35:20.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4776 16 3599 3583 0.0468813 0 - 0.36018 2024-04-22T07:35:21.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4777 16 3599 3583 0.0468715 0 - 0.36018 2024-04-22T07:35:22.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4778 16 3599 3583 0.0468617 0 - 0.36018 2024-04-22T07:35:23.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4779 16 3599 3583 0.0468519 0 - 0.36018 2024-04-22T07:35:24.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:35:24.139068+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:35:24.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:35:24.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4780 16 3599 3583 0.0468421 0 - 0.36018 2024-04-22T07:35:25.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4781 16 3599 3583 0.0468323 0 - 0.36018 2024-04-22T07:35:26.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4782 16 3599 3583 0.0468225 0 - 0.36018 2024-04-22T07:35:27.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4783 16 3599 3583 0.0468127 0 - 0.36018 2024-04-22T07:35:28.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4784 16 3599 3583 0.0468029 0 - 0.36018 2024-04-22T07:35:29.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4785 16 3599 3583 0.0467931 0 - 0.36018 2024-04-22T07:35:30.137 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4786 16 3599 3583 0.0467834 0 - 0.36018 2024-04-22T07:35:31.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4787 16 3599 3583 0.0467736 0 - 0.36018 2024-04-22T07:35:32.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4788 16 3599 3583 0.0467638 0 - 0.36018 2024-04-22T07:35:33.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4789 16 3599 3583 0.0467541 0 - 0.36018 2024-04-22T07:35:34.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4790 16 3599 3583 0.0467443 0 - 0.36018 2024-04-22T07:35:35.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4791 16 3599 3583 0.0467345 0 - 0.36018 2024-04-22T07:35:36.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4792 16 3599 3583 0.0467248 0 - 0.36018 2024-04-22T07:35:37.138 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4793 16 3599 3583 0.046715 0 - 0.36018 2024-04-22T07:35:38.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4794 16 3599 3583 0.0467053 0 - 0.36018 2024-04-22T07:35:39.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4795 16 3599 3583 0.0466956 0 - 0.36018 2024-04-22T07:35:40.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4796 16 3599 3583 0.0466858 0 - 0.36018 2024-04-22T07:35:41.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4797 16 3599 3583 0.0466761 0 - 0.36018 2024-04-22T07:35:42.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4798 16 3599 3583 0.0466664 0 - 0.36018 2024-04-22T07:35:43.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4799 16 3599 3583 0.0466566 0 - 0.36018 2024-04-22T07:35:44.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:35:44.141968+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:35:44.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:35:44.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4800 16 3599 3583 0.0466469 0 - 0.36018 2024-04-22T07:35:45.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4801 16 3599 3583 0.0466372 0 - 0.36018 2024-04-22T07:35:46.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4802 16 3599 3583 0.0466275 0 - 0.36018 2024-04-22T07:35:47.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4803 16 3599 3583 0.0466178 0 - 0.36018 2024-04-22T07:35:48.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4804 16 3599 3583 0.0466081 0 - 0.36018 2024-04-22T07:35:49.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4805 16 3599 3583 0.0465984 0 - 0.36018 2024-04-22T07:35:50.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4806 16 3599 3583 0.0465887 0 - 0.36018 2024-04-22T07:35:51.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4807 16 3599 3583 0.046579 0 - 0.36018 2024-04-22T07:35:52.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4808 16 3599 3583 0.0465693 0 - 0.36018 2024-04-22T07:35:53.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4809 16 3599 3583 0.0465596 0 - 0.36018 2024-04-22T07:35:54.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4810 16 3599 3583 0.0465499 0 - 0.36018 2024-04-22T07:35:55.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4811 16 3599 3583 0.0465403 0 - 0.36018 2024-04-22T07:35:56.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4812 16 3599 3583 0.0465306 0 - 0.36018 2024-04-22T07:35:57.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4813 16 3599 3583 0.0465209 0 - 0.36018 2024-04-22T07:35:58.142 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4814 16 3599 3583 0.0465113 0 - 0.36018 2024-04-22T07:35:59.142 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4815 16 3599 3583 0.0465016 0 - 0.36018 2024-04-22T07:36:00.142 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4816 16 3599 3583 0.0464919 0 - 0.36018 2024-04-22T07:36:01.142 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4817 16 3599 3583 0.0464823 0 - 0.36018 2024-04-22T07:36:02.142 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4818 16 3599 3583 0.0464726 0 - 0.36018 2024-04-22T07:36:03.142 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4819 16 3599 3583 0.046463 0 - 0.36018 2024-04-22T07:36:04.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:36:04.144924+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:36:04.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:36:04.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4820 16 3599 3583 0.0464534 0 - 0.36018 2024-04-22T07:36:05.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4821 16 3599 3583 0.0464437 0 - 0.36018 2024-04-22T07:36:06.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4822 16 3599 3583 0.0464341 0 - 0.36018 2024-04-22T07:36:07.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4823 16 3599 3583 0.0464245 0 - 0.36018 2024-04-22T07:36:08.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4824 16 3599 3583 0.0464148 0 - 0.36018 2024-04-22T07:36:09.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4825 16 3599 3583 0.0464052 0 - 0.36018 2024-04-22T07:36:10.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4826 16 3599 3583 0.0463956 0 - 0.36018 2024-04-22T07:36:11.143 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4827 16 3599 3583 0.046386 0 - 0.36018 2024-04-22T07:36:12.144 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4828 16 3599 3583 0.0463764 0 - 0.36018 2024-04-22T07:36:13.144 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4829 16 3599 3583 0.0463668 0 - 0.36018 2024-04-22T07:36:14.144 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4830 16 3599 3583 0.0463572 0 - 0.36018 2024-04-22T07:36:15.144 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4831 16 3599 3583 0.0463476 0 - 0.36018 2024-04-22T07:36:16.144 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4832 16 3599 3583 0.046338 0 - 0.36018 2024-04-22T07:36:17.144 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4833 16 3599 3583 0.0463284 0 - 0.36018 2024-04-22T07:36:18.145 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4834 16 3599 3583 0.0463188 0 - 0.36018 2024-04-22T07:36:19.145 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4835 16 3599 3583 0.0463092 0 - 0.36018 2024-04-22T07:36:20.145 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4836 16 3599 3583 0.0462997 0 - 0.36018 2024-04-22T07:36:21.145 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4837 16 3599 3583 0.0462901 0 - 0.36018 2024-04-22T07:36:22.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4838 16 3599 3583 0.0462805 0 - 0.36018 2024-04-22T07:36:23.145 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4839 16 3599 3583 0.046271 0 - 0.36018 2024-04-22T07:36:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:36:23.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:36:24.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:36:24.147875+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:36:24.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:36:24.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4840 16 3599 3583 0.0462614 0 - 0.36018 2024-04-22T07:36:25.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4841 16 3599 3583 0.0462518 0 - 0.36018 2024-04-22T07:36:25.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T07:36:25.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:36:25.381+0000 7ff14906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:36:26.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4842 16 3599 3583 0.0462423 0 - 0.36018 2024-04-22T07:36:27.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4843 16 3599 3583 0.0462327 0 - 0.36018 2024-04-22T07:36:28.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4844 16 3599 3583 0.0462232 0 - 0.36018 2024-04-22T07:36:29.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4845 16 3599 3583 0.0462137 0 - 0.36018 2024-04-22T07:36:30.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4846 16 3599 3583 0.0462041 0 - 0.36018 2024-04-22T07:36:31.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4847 16 3599 3583 0.0461946 0 - 0.36018 2024-04-22T07:36:32.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4848 16 3599 3583 0.0461851 0 - 0.36018 2024-04-22T07:36:33.147 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4849 16 3599 3583 0.0461755 0 - 0.36018 2024-04-22T07:36:34.147 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4850 16 3599 3583 0.046166 0 - 0.36018 2024-04-22T07:36:35.147 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4851 16 3599 3583 0.0461565 0 - 0.36018 2024-04-22T07:36:36.147 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4852 16 3599 3583 0.046147 0 - 0.36018 2024-04-22T07:36:37.147 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4853 16 3599 3583 0.0461375 0 - 0.36018 2024-04-22T07:36:38.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4854 16 3599 3583 0.046128 0 - 0.36018 2024-04-22T07:36:39.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4855 16 3599 3583 0.0461185 0 - 0.36018 2024-04-22T07:36:40.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4856 16 3599 3583 0.046109 0 - 0.36018 2024-04-22T07:36:41.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4857 16 3599 3583 0.0460995 0 - 0.36018 2024-04-22T07:36:42.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4858 16 3599 3583 0.04609 0 - 0.36018 2024-04-22T07:36:43.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4859 16 3599 3583 0.0460805 0 - 0.36018 2024-04-22T07:36:44.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:36:44.150751+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:36:44.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:36:44.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4860 16 3599 3583 0.046071 0 - 0.36018 2024-04-22T07:36:45.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4861 16 3599 3583 0.0460616 0 - 0.36018 2024-04-22T07:36:46.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4862 16 3599 3583 0.0460521 0 - 0.36018 2024-04-22T07:36:47.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4863 16 3599 3583 0.0460426 0 - 0.36018 2024-04-22T07:36:48.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4864 16 3599 3583 0.0460331 0 - 0.36018 2024-04-22T07:36:49.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4865 16 3599 3583 0.0460237 0 - 0.36018 2024-04-22T07:36:50.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4866 16 3599 3583 0.0460142 0 - 0.36018 2024-04-22T07:36:51.149 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4867 16 3599 3583 0.0460048 0 - 0.36018 2024-04-22T07:36:52.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4868 16 3599 3583 0.0459953 0 - 0.36018 2024-04-22T07:36:53.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4869 16 3599 3583 0.0459859 0 - 0.36018 2024-04-22T07:36:54.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4870 16 3599 3583 0.0459764 0 - 0.36018 2024-04-22T07:36:55.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4871 16 3599 3583 0.045967 0 - 0.36018 2024-04-22T07:36:56.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4872 16 3599 3583 0.0459576 0 - 0.36018 2024-04-22T07:36:57.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4873 16 3599 3583 0.0459481 0 - 0.36018 2024-04-22T07:36:58.150 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4874 16 3599 3583 0.0459387 0 - 0.36018 2024-04-22T07:36:59.151 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4875 16 3599 3583 0.0459293 0 - 0.36018 2024-04-22T07:37:00.151 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4876 16 3599 3583 0.0459199 0 - 0.36018 2024-04-22T07:37:00.573 ERROR:teuthology.run_tasks:Manager failed: radosbench Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds 2024-04-22T07:37:00.748 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-22T07:37:00.758 INFO:tasks.thrashosds:joining thrashosds 2024-04-22T07:37:00.758 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-22T07:37:01.151 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4877 16 3599 3583 0.0459104 0 - 0.36018 2024-04-22T07:37:02.151 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4878 16 3599 3583 0.045901 0 - 0.36018 2024-04-22T07:37:03.151 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4879 16 3599 3583 0.0458916 0 - 0.36018 2024-04-22T07:37:04.151 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:37:04.153652+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:37:04.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:37:04.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4880 16 3599 3583 0.0458822 0 - 0.36018 2024-04-22T07:37:05.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4881 16 3599 3583 0.0458728 0 - 0.36018 2024-04-22T07:37:06.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4882 16 3599 3583 0.0458634 0 - 0.36018 2024-04-22T07:37:07.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4883 16 3599 3583 0.045854 0 - 0.36018 2024-04-22T07:37:08.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4884 16 3599 3583 0.0458446 0 - 0.36018 2024-04-22T07:37:09.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4885 16 3599 3583 0.0458353 0 - 0.36018 2024-04-22T07:37:10.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4886 16 3599 3583 0.0458259 0 - 0.36018 2024-04-22T07:37:11.152 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4887 16 3599 3583 0.0458165 0 - 0.36018 2024-04-22T07:37:12.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4888 16 3599 3583 0.0458071 0 - 0.36018 2024-04-22T07:37:13.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4889 16 3599 3583 0.0457977 0 - 0.36018 2024-04-22T07:37:14.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4890 16 3599 3583 0.0457884 0 - 0.36018 2024-04-22T07:37:15.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4891 16 3599 3583 0.045779 0 - 0.36018 2024-04-22T07:37:16.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4892 16 3599 3583 0.0457697 0 - 0.36018 2024-04-22T07:37:17.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4893 16 3599 3583 0.0457603 0 - 0.36018 2024-04-22T07:37:18.153 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4894 16 3599 3583 0.045751 0 - 0.36018 2024-04-22T07:37:19.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4895 16 3599 3583 0.0457416 0 - 0.36018 2024-04-22T07:37:20.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4896 16 3599 3583 0.0457323 0 - 0.36018 2024-04-22T07:37:21.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4897 16 3599 3583 0.0457229 0 - 0.36018 2024-04-22T07:37:22.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4898 16 3599 3583 0.0457136 0 - 0.36018 2024-04-22T07:37:23.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4899 16 3599 3583 0.0457043 0 - 0.36018 2024-04-22T07:37:24.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:37:24.156535+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:37:24.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:37:24.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4900 16 3599 3583 0.0456949 0 - 0.36018 2024-04-22T07:37:25.154 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4901 16 3599 3583 0.0456856 0 - 0.36018 2024-04-22T07:37:26.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4902 16 3599 3583 0.0456763 0 - 0.36018 2024-04-22T07:37:27.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4903 16 3599 3583 0.045667 0 - 0.36018 2024-04-22T07:37:28.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4904 16 3599 3583 0.0456577 0 - 0.36018 2024-04-22T07:37:29.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4905 16 3599 3583 0.0456484 0 - 0.36018 2024-04-22T07:37:30.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4906 16 3599 3583 0.0456391 0 - 0.36018 2024-04-22T07:37:31.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4907 16 3599 3583 0.0456298 0 - 0.36018 2024-04-22T07:37:32.155 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4908 16 3599 3583 0.0456205 0 - 0.36018 2024-04-22T07:37:33.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4909 16 3599 3583 0.0456112 0 - 0.36018 2024-04-22T07:37:34.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4910 16 3599 3583 0.0456019 0 - 0.36018 2024-04-22T07:37:35.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4911 16 3599 3583 0.0455926 0 - 0.36018 2024-04-22T07:37:36.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4912 16 3599 3583 0.0455833 0 - 0.36018 2024-04-22T07:37:37.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4913 16 3599 3583 0.045574 0 - 0.36018 2024-04-22T07:37:38.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4914 16 3599 3583 0.0455648 0 - 0.36018 2024-04-22T07:37:39.156 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4915 16 3599 3583 0.0455555 0 - 0.36018 2024-04-22T07:37:40.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4916 16 3599 3583 0.0455462 0 - 0.36018 2024-04-22T07:37:41.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4917 16 3599 3583 0.045537 0 - 0.36018 2024-04-22T07:37:42.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4918 16 3599 3583 0.0455277 0 - 0.36018 2024-04-22T07:37:43.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4919 16 3599 3583 0.0455184 0 - 0.36018 2024-04-22T07:37:44.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:37:44.159360+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:37:44.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:37:44.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4920 16 3599 3583 0.0455092 0 - 0.36018 2024-04-22T07:37:45.157 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4921 16 3599 3583 0.0454999 0 - 0.36018 2024-04-22T07:37:46.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4922 16 3599 3583 0.0454907 0 - 0.36018 2024-04-22T07:37:47.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4923 16 3599 3583 0.0454815 0 - 0.36018 2024-04-22T07:37:48.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4924 16 3599 3583 0.0454722 0 - 0.36018 2024-04-22T07:37:49.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4925 16 3599 3583 0.045463 0 - 0.36018 2024-04-22T07:37:50.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4926 16 3599 3583 0.0454538 0 - 0.36018 2024-04-22T07:37:51.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4927 16 3599 3583 0.0454445 0 - 0.36018 2024-04-22T07:37:52.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4928 16 3599 3583 0.0454353 0 - 0.36018 2024-04-22T07:37:53.158 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4929 16 3599 3583 0.0454261 0 - 0.36018 2024-04-22T07:37:54.159 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4930 16 3599 3583 0.0454169 0 - 0.36018 2024-04-22T07:37:55.159 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4931 16 3599 3583 0.0454077 0 - 0.36018 2024-04-22T07:37:56.159 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4932 16 3599 3583 0.0453985 0 - 0.36018 2024-04-22T07:37:57.159 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4933 16 3599 3583 0.0453893 0 - 0.36018 2024-04-22T07:37:58.159 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4934 16 3599 3583 0.0453801 0 - 0.36018 2024-04-22T07:37:59.159 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4935 16 3599 3583 0.0453709 0 - 0.36018 2024-04-22T07:38:00.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4936 16 3599 3583 0.0453617 0 - 0.36018 2024-04-22T07:38:01.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4937 16 3599 3583 0.0453525 0 - 0.36018 2024-04-22T07:38:02.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4938 16 3599 3583 0.0453433 0 - 0.36018 2024-04-22T07:38:03.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4939 16 3599 3583 0.0453341 0 - 0.36018 2024-04-22T07:38:04.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:38:04.162185+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:38:04.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:38:04.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4940 16 3599 3583 0.0453249 0 - 0.36018 2024-04-22T07:38:05.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4941 16 3599 3583 0.0453158 0 - 0.36018 2024-04-22T07:38:06.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4942 16 3599 3583 0.0453066 0 - 0.36018 2024-04-22T07:38:07.160 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4943 16 3599 3583 0.0452974 0 - 0.36018 2024-04-22T07:38:08.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4944 16 3599 3583 0.0452883 0 - 0.36018 2024-04-22T07:38:09.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4945 16 3599 3583 0.0452791 0 - 0.36018 2024-04-22T07:38:10.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4946 16 3599 3583 0.04527 0 - 0.36018 2024-04-22T07:38:11.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4947 16 3599 3583 0.0452608 0 - 0.36018 2024-04-22T07:38:12.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4948 16 3599 3583 0.0452517 0 - 0.36018 2024-04-22T07:38:13.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4949 16 3599 3583 0.0452425 0 - 0.36018 2024-04-22T07:38:14.161 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4950 16 3599 3583 0.0452334 0 - 0.36018 2024-04-22T07:38:15.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4951 16 3599 3583 0.0452242 0 - 0.36018 2024-04-22T07:38:16.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4952 16 3599 3583 0.0452151 0 - 0.36018 2024-04-22T07:38:17.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4953 16 3599 3583 0.045206 0 - 0.36018 2024-04-22T07:38:18.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4954 16 3599 3583 0.0451969 0 - 0.36018 2024-04-22T07:38:19.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4955 16 3599 3583 0.0451877 0 - 0.36018 2024-04-22T07:38:20.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4956 16 3599 3583 0.0451786 0 - 0.36018 2024-04-22T07:38:21.162 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4957 16 3599 3583 0.0451695 0 - 0.36018 2024-04-22T07:38:22.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4958 16 3599 3583 0.0451604 0 - 0.36018 2024-04-22T07:38:23.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4959 16 3599 3583 0.0451513 0 - 0.36018 2024-04-22T07:38:24.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:38:24.165057+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:38:24.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:38:24.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4960 16 3599 3583 0.0451422 0 - 0.36018 2024-04-22T07:38:25.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4961 16 3599 3583 0.0451331 0 - 0.36018 2024-04-22T07:38:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:38:25.294 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T07:38:26.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4962 16 3599 3583 0.045124 0 - 0.36018 2024-04-22T07:38:27.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4963 16 3599 3583 0.0451149 0 - 0.36018 2024-04-22T07:38:27.296 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T07:38:27.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T07:38:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:38:27.419+0000 7fa3e7154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:38:28.163 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4964 16 3599 3583 0.0451058 0 - 0.36018 2024-04-22T07:38:29.164 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4965 16 3599 3583 0.0450967 0 - 0.36018 2024-04-22T07:38:30.164 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4966 16 3599 3583 0.0450876 0 - 0.36018 2024-04-22T07:38:31.164 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4967 16 3599 3583 0.0450786 0 - 0.36018 2024-04-22T07:38:32.164 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4968 16 3599 3583 0.0450695 0 - 0.36018 2024-04-22T07:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4969 16 3599 3583 0.0450604 0 - 0.36018 2024-04-22T07:38:34.164 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4970 16 3599 3583 0.0450513 0 - 0.36018 2024-04-22T07:38:35.165 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4971 16 3599 3583 0.0450423 0 - 0.36018 2024-04-22T07:38:36.165 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4972 16 3599 3583 0.0450332 0 - 0.36018 2024-04-22T07:38:37.165 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4973 16 3599 3583 0.0450242 0 - 0.36018 2024-04-22T07:38:38.165 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4974 16 3599 3583 0.0450151 0 - 0.36018 2024-04-22T07:38:39.165 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4975 16 3599 3583 0.0450061 0 - 0.36018 2024-04-22T07:38:40.165 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4976 16 3599 3583 0.044997 0 - 0.36018 2024-04-22T07:38:41.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4977 16 3599 3583 0.044988 0 - 0.36018 2024-04-22T07:38:42.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4978 16 3599 3583 0.0449789 0 - 0.36018 2024-04-22T07:38:43.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4979 16 3599 3583 0.0449699 0 - 0.36018 2024-04-22T07:38:44.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:38:44.167912+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:38:44.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:38:44.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4980 16 3599 3583 0.0449609 0 - 0.36018 2024-04-22T07:38:45.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4981 16 3599 3583 0.0449519 0 - 0.36018 2024-04-22T07:38:46.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4982 16 3599 3583 0.0449428 0 - 0.36018 2024-04-22T07:38:47.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4983 16 3599 3583 0.0449338 0 - 0.36018 2024-04-22T07:38:48.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4984 16 3599 3583 0.0449248 0 - 0.36018 2024-04-22T07:38:49.166 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4985 16 3599 3583 0.0449158 0 - 0.36018 2024-04-22T07:38:50.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4986 16 3599 3583 0.0449068 0 - 0.36018 2024-04-22T07:38:51.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4987 16 3599 3583 0.0448978 0 - 0.36018 2024-04-22T07:38:52.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4988 16 3599 3583 0.0448888 0 - 0.36018 2024-04-22T07:38:53.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4989 16 3599 3583 0.0448798 0 - 0.36018 2024-04-22T07:38:54.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4990 16 3599 3583 0.0448708 0 - 0.36018 2024-04-22T07:38:55.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4991 16 3599 3583 0.0448618 0 - 0.36018 2024-04-22T07:38:56.167 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4992 16 3599 3583 0.0448528 0 - 0.36018 2024-04-22T07:38:57.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4993 16 3599 3583 0.0448438 0 - 0.36018 2024-04-22T07:38:58.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4994 16 3599 3583 0.0448348 0 - 0.36018 2024-04-22T07:38:59.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4995 16 3599 3583 0.0448259 0 - 0.36018 2024-04-22T07:39:00.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4996 16 3599 3583 0.0448169 0 - 0.36018 2024-04-22T07:39:01.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4997 16 3599 3583 0.0448079 0 - 0.36018 2024-04-22T07:39:02.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4998 16 3599 3583 0.044799 0 - 0.36018 2024-04-22T07:39:03.168 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4999 16 3599 3583 0.04479 0 - 0.36018 2024-04-22T07:39:04.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:39:04.170674+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:39:04.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:39:04.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5000 16 3599 3583 0.044781 0 - 0.36018 2024-04-22T07:39:05.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5001 16 3599 3583 0.0447721 0 - 0.36018 2024-04-22T07:39:06.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5002 16 3599 3583 0.0447631 0 - 0.36018 2024-04-22T07:39:07.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5003 16 3599 3583 0.0447542 0 - 0.36018 2024-04-22T07:39:08.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5004 16 3599 3583 0.0447452 0 - 0.36018 2024-04-22T07:39:09.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5005 16 3599 3583 0.0447363 0 - 0.36018 2024-04-22T07:39:10.169 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5006 16 3599 3583 0.0447274 0 - 0.36018 2024-04-22T07:39:11.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5007 16 3599 3583 0.0447184 0 - 0.36018 2024-04-22T07:39:12.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5008 16 3599 3583 0.0447095 0 - 0.36018 2024-04-22T07:39:13.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5009 16 3599 3583 0.0447006 0 - 0.36018 2024-04-22T07:39:14.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5010 16 3599 3583 0.0446917 0 - 0.36018 2024-04-22T07:39:15.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5011 16 3599 3583 0.0446827 0 - 0.36018 2024-04-22T07:39:16.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5012 16 3599 3583 0.0446738 0 - 0.36018 2024-04-22T07:39:17.170 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5013 16 3599 3583 0.0446649 0 - 0.36018 2024-04-22T07:39:18.171 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5014 16 3599 3583 0.044656 0 - 0.36018 2024-04-22T07:39:19.171 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5015 16 3599 3583 0.0446471 0 - 0.36018 2024-04-22T07:39:20.171 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5016 16 3599 3583 0.0446382 0 - 0.36018 2024-04-22T07:39:21.171 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5017 16 3599 3583 0.0446293 0 - 0.36018 2024-04-22T07:39:22.171 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5018 16 3599 3583 0.0446204 0 - 0.36018 2024-04-22T07:39:23.171 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5019 16 3599 3583 0.0446115 0 - 0.36018 2024-04-22T07:39:24.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:39:24.173611+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:39:24.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:39:24.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5020 16 3599 3583 0.0446026 0 - 0.36018 2024-04-22T07:39:25.172 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5021 16 3599 3583 0.0445937 0 - 0.36018 2024-04-22T07:39:26.172 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5022 16 3599 3583 0.0445849 0 - 0.36018 2024-04-22T07:39:27.172 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5023 16 3599 3583 0.044576 0 - 0.36018 2024-04-22T07:39:28.172 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5024 16 3599 3583 0.0445671 0 - 0.36018 2024-04-22T07:39:29.172 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5025 16 3599 3583 0.0445583 0 - 0.36018 2024-04-22T07:39:30.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5026 16 3599 3583 0.0445494 0 - 0.36018 2024-04-22T07:39:31.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5027 16 3599 3583 0.0445405 0 - 0.36018 2024-04-22T07:39:32.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5028 16 3599 3583 0.0445317 0 - 0.36018 2024-04-22T07:39:33.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5029 16 3599 3583 0.0445228 0 - 0.36018 2024-04-22T07:39:34.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5030 16 3599 3583 0.044514 0 - 0.36018 2024-04-22T07:39:35.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5031 16 3599 3583 0.0445051 0 - 0.36018 2024-04-22T07:39:36.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5032 16 3599 3583 0.0444963 0 - 0.36018 2024-04-22T07:39:37.173 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5033 16 3599 3583 0.0444874 0 - 0.36018 2024-04-22T07:39:38.174 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5034 16 3599 3583 0.0444786 0 - 0.36018 2024-04-22T07:39:39.174 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5035 16 3599 3583 0.0444698 0 - 0.36018 2024-04-22T07:39:40.174 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5036 16 3599 3583 0.0444609 0 - 0.36018 2024-04-22T07:39:41.174 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5037 16 3599 3583 0.0444521 0 - 0.36018 2024-04-22T07:39:42.174 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5038 16 3599 3583 0.0444433 0 - 0.36018 2024-04-22T07:39:43.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5039 16 3599 3583 0.0444345 0 - 0.36018 2024-04-22T07:39:44.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:39:44.176557+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:39:44.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:39:44.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5040 16 3599 3583 0.0444256 0 - 0.36018 2024-04-22T07:39:45.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5041 16 3599 3583 0.0444168 0 - 0.36018 2024-04-22T07:39:46.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5042 16 3599 3583 0.044408 0 - 0.36018 2024-04-22T07:39:47.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5043 16 3599 3583 0.0443992 0 - 0.36018 2024-04-22T07:39:48.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5044 16 3599 3583 0.0443904 0 - 0.36018 2024-04-22T07:39:49.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5045 16 3599 3583 0.0443816 0 - 0.36018 2024-04-22T07:39:50.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5046 16 3599 3583 0.0443728 0 - 0.36018 2024-04-22T07:39:51.175 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5047 16 3599 3583 0.044364 0 - 0.36018 2024-04-22T07:39:52.176 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5048 16 3599 3583 0.0443552 0 - 0.36018 2024-04-22T07:39:53.176 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5049 16 3599 3583 0.0443464 0 - 0.36018 2024-04-22T07:39:54.176 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5050 16 3599 3583 0.0443377 0 - 0.36018 2024-04-22T07:39:55.176 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5051 16 3599 3583 0.0443289 0 - 0.36018 2024-04-22T07:39:56.176 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5052 16 3599 3583 0.0443201 0 - 0.36018 2024-04-22T07:39:57.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5053 16 3599 3583 0.0443113 0 - 0.36018 2024-04-22T07:39:58.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5054 16 3599 3583 0.0443026 0 - 0.36018 2024-04-22T07:39:59.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5055 16 3599 3583 0.0442938 0 - 0.36018 2024-04-22T07:40:00.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5056 16 3599 3583 0.044285 0 - 0.36018 2024-04-22T07:40:01.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5057 16 3599 3583 0.0442763 0 - 0.36018 2024-04-22T07:40:02.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5058 16 3599 3583 0.0442675 0 - 0.36018 2024-04-22T07:40:03.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5059 16 3599 3583 0.0442588 0 - 0.36018 2024-04-22T07:40:04.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:40:04.179470+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:40:04.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:40:04.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5060 16 3599 3583 0.04425 0 - 0.36018 2024-04-22T07:40:05.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5061 16 3599 3583 0.0442413 0 - 0.36018 2024-04-22T07:40:06.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5062 16 3599 3583 0.0442326 0 - 0.36018 2024-04-22T07:40:07.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5063 16 3599 3583 0.0442238 0 - 0.36018 2024-04-22T07:40:08.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5064 16 3599 3583 0.0442151 0 - 0.36018 2024-04-22T07:40:09.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5065 16 3599 3583 0.0442064 0 - 0.36018 2024-04-22T07:40:10.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5066 16 3599 3583 0.0441976 0 - 0.36018 2024-04-22T07:40:11.178 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5067 16 3599 3583 0.0441889 0 - 0.36018 2024-04-22T07:40:12.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5068 16 3599 3583 0.0441802 0 - 0.36018 2024-04-22T07:40:13.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5069 16 3599 3583 0.0441715 0 - 0.36018 2024-04-22T07:40:14.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5070 16 3599 3583 0.0441628 0 - 0.36018 2024-04-22T07:40:15.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5071 16 3599 3583 0.0441541 0 - 0.36018 2024-04-22T07:40:16.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5072 16 3599 3583 0.0441453 0 - 0.36018 2024-04-22T07:40:17.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5073 16 3599 3583 0.0441366 0 - 0.36018 2024-04-22T07:40:18.179 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5074 16 3599 3583 0.0441279 0 - 0.36018 2024-04-22T07:40:19.180 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5075 16 3599 3583 0.0441193 0 - 0.36018 2024-04-22T07:40:20.180 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5076 16 3599 3583 0.0441106 0 - 0.36018 2024-04-22T07:40:21.180 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5077 16 3599 3583 0.0441019 0 - 0.36018 2024-04-22T07:40:22.180 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5078 16 3599 3583 0.0440932 0 - 0.36018 2024-04-22T07:40:23.180 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5079 16 3599 3583 0.0440845 0 - 0.36018 2024-04-22T07:40:24.180 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:40:24.182376+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:40:24.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:40:24.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5080 16 3599 3583 0.0440758 0 - 0.36018 2024-04-22T07:40:25.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5081 16 3599 3583 0.0440672 0 - 0.36018 2024-04-22T07:40:26.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5082 16 3599 3583 0.0440585 0 - 0.36018 2024-04-22T07:40:27.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5083 16 3599 3583 0.0440498 0 - 0.36018 2024-04-22T07:40:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:40:27.333 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-22T07:40:27.383 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T07:40:27.393 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T07:40:27.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T07:40:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-22T07:40:27.515+0000 7f66f240f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T07:40:28.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5084 16 3599 3583 0.0440412 0 - 0.36018 2024-04-22T07:40:29.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5085 16 3599 3583 0.0440325 0 - 0.36018 2024-04-22T07:40:30.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5086 16 3599 3583 0.0440238 0 - 0.36018 2024-04-22T07:40:31.181 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5087 16 3599 3583 0.0440152 0 - 0.36018 2024-04-22T07:40:32.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5088 16 3599 3583 0.0440065 0 - 0.36018 2024-04-22T07:40:33.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5089 16 3599 3583 0.0439979 0 - 0.36018 2024-04-22T07:40:34.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5090 16 3599 3583 0.0439892 0 - 0.36018 2024-04-22T07:40:35.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5091 16 3599 3583 0.0439806 0 - 0.36018 2024-04-22T07:40:36.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5092 16 3599 3583 0.043972 0 - 0.36018 2024-04-22T07:40:37.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5093 16 3599 3583 0.0439633 0 - 0.36018 2024-04-22T07:40:38.182 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5094 16 3599 3583 0.0439547 0 - 0.36018 2024-04-22T07:40:39.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5095 16 3599 3583 0.0439461 0 - 0.36018 2024-04-22T07:40:40.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5096 16 3599 3583 0.0439374 0 - 0.36018 2024-04-22T07:40:41.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5097 16 3599 3583 0.0439288 0 - 0.36018 2024-04-22T07:40:42.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5098 16 3599 3583 0.0439202 0 - 0.36018 2024-04-22T07:40:43.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5099 16 3599 3583 0.0439116 0 - 0.36018 2024-04-22T07:40:44.183 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:40:44.185219+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:40:44.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:40:44.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5100 16 3599 3583 0.043903 0 - 0.36018 2024-04-22T07:40:45.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5101 16 3599 3583 0.0438944 0 - 0.36018 2024-04-22T07:40:46.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5102 16 3599 3583 0.0438858 0 - 0.36018 2024-04-22T07:40:47.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5103 16 3599 3583 0.0438772 0 - 0.36018 2024-04-22T07:40:48.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5104 16 3599 3583 0.0438686 0 - 0.36018 2024-04-22T07:40:49.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5105 16 3599 3583 0.04386 0 - 0.36018 2024-04-22T07:40:50.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5106 16 3599 3583 0.0438514 0 - 0.36018 2024-04-22T07:40:51.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5107 16 3599 3583 0.0438428 0 - 0.36018 2024-04-22T07:40:52.184 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5108 16 3599 3583 0.0438342 0 - 0.36018 2024-04-22T07:40:53.185 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5109 16 3599 3583 0.0438256 0 - 0.36018 2024-04-22T07:40:54.185 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5110 16 3599 3583 0.0438171 0 - 0.36018 2024-04-22T07:40:55.185 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5111 16 3599 3583 0.0438085 0 - 0.36018 2024-04-22T07:40:56.185 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5112 16 3599 3583 0.0437999 0 - 0.36018 2024-04-22T07:40:57.185 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5113 16 3599 3583 0.0437914 0 - 0.36018 2024-04-22T07:40:58.185 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5114 16 3599 3583 0.0437828 0 - 0.36018 2024-04-22T07:40:59.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5115 16 3599 3583 0.0437742 0 - 0.36018 2024-04-22T07:41:00.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5116 16 3599 3583 0.0437657 0 - 0.36018 2024-04-22T07:41:01.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5117 16 3599 3583 0.0437571 0 - 0.36018 2024-04-22T07:41:02.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5118 16 3599 3583 0.0437486 0 - 0.36018 2024-04-22T07:41:03.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5119 16 3599 3583 0.04374 0 - 0.36018 2024-04-22T07:41:04.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:41:04.188241+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:41:04.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:41:04.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5120 16 3599 3583 0.0437315 0 - 0.36018 2024-04-22T07:41:05.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5121 16 3599 3583 0.0437229 0 - 0.36018 2024-04-22T07:41:06.186 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5122 16 3599 3583 0.0437144 0 - 0.36018 2024-04-22T07:41:07.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5123 16 3599 3583 0.0437059 0 - 0.36018 2024-04-22T07:41:08.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5124 16 3599 3583 0.0436973 0 - 0.36018 2024-04-22T07:41:09.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5125 16 3599 3583 0.0436888 0 - 0.36018 2024-04-22T07:41:10.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5126 16 3599 3583 0.0436803 0 - 0.36018 2024-04-22T07:41:11.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5127 16 3599 3583 0.0436718 0 - 0.36018 2024-04-22T07:41:12.187 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5128 16 3599 3583 0.0436633 0 - 0.36018 2024-04-22T07:41:13.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5129 16 3599 3583 0.0436547 0 - 0.36018 2024-04-22T07:41:14.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5130 16 3599 3583 0.0436462 0 - 0.36018 2024-04-22T07:41:15.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5131 16 3599 3583 0.0436377 0 - 0.36018 2024-04-22T07:41:16.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5132 16 3599 3583 0.0436292 0 - 0.36018 2024-04-22T07:41:17.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5133 16 3599 3583 0.0436207 0 - 0.36018 2024-04-22T07:41:18.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5134 16 3599 3583 0.0436122 0 - 0.36018 2024-04-22T07:41:19.188 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5135 16 3599 3583 0.0436037 0 - 0.36018 2024-04-22T07:41:20.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5136 16 3599 3583 0.0435952 0 - 0.36018 2024-04-22T07:41:21.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5137 16 3599 3583 0.0435868 0 - 0.36018 2024-04-22T07:41:22.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5138 16 3599 3583 0.0435783 0 - 0.36018 2024-04-22T07:41:23.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5139 16 3599 3583 0.0435698 0 - 0.36018 2024-04-22T07:41:24.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:41:24.191132+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:41:24.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:41:24.190 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5140 16 3599 3583 0.0435613 0 - 0.36018 2024-04-22T07:41:25.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5141 16 3599 3583 0.0435528 0 - 0.36018 2024-04-22T07:41:26.189 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5142 16 3599 3583 0.0435444 0 - 0.36018 2024-04-22T07:41:27.190 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5143 16 3599 3583 0.0435359 0 - 0.36018 2024-04-22T07:41:28.190 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5144 16 3599 3583 0.0435274 0 - 0.36018 2024-04-22T07:41:29.190 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5145 16 3599 3583 0.043519 0 - 0.36018 2024-04-22T07:41:30.190 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5146 16 3599 3583 0.0435105 0 - 0.36018 2024-04-22T07:41:31.190 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5147 16 3599 3583 0.0435021 0 - 0.36018 2024-04-22T07:41:32.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5148 16 3599 3583 0.0434936 0 - 0.36018 2024-04-22T07:41:33.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5149 16 3599 3583 0.0434852 0 - 0.36018 2024-04-22T07:41:34.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5150 16 3599 3583 0.0434767 0 - 0.36018 2024-04-22T07:41:35.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5151 16 3599 3583 0.0434683 0 - 0.36018 2024-04-22T07:41:36.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5152 16 3599 3583 0.0434599 0 - 0.36018 2024-04-22T07:41:37.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5153 16 3599 3583 0.0434514 0 - 0.36018 2024-04-22T07:41:38.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5154 16 3599 3583 0.043443 0 - 0.36018 2024-04-22T07:41:39.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5155 16 3599 3583 0.0434346 0 - 0.36018 2024-04-22T07:41:40.191 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5156 16 3599 3583 0.0434261 0 - 0.36018 2024-04-22T07:41:41.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5157 16 3599 3583 0.0434177 0 - 0.36018 2024-04-22T07:41:42.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5158 16 3599 3583 0.0434093 0 - 0.36018 2024-04-22T07:41:43.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5159 16 3599 3583 0.0434009 0 - 0.36018 2024-04-22T07:41:44.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:41:44.194080+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:41:44.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:41:44.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5160 16 3599 3583 0.0433925 0 - 0.36018 2024-04-22T07:41:45.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5161 16 3599 3583 0.0433841 0 - 0.36018 2024-04-22T07:41:46.192 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5162 16 3599 3583 0.0433757 0 - 0.36018 2024-04-22T07:41:47.193 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5163 16 3599 3583 0.0433673 0 - 0.36018 2024-04-22T07:41:48.193 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5164 16 3599 3583 0.0433589 0 - 0.36018 2024-04-22T07:41:49.193 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5165 16 3599 3583 0.0433505 0 - 0.36018 2024-04-22T07:41:50.193 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5166 16 3599 3583 0.0433421 0 - 0.36018 2024-04-22T07:41:51.193 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5167 16 3599 3583 0.0433337 0 - 0.36018 2024-04-22T07:41:52.193 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5168 16 3599 3583 0.0433253 0 - 0.36018 2024-04-22T07:41:53.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5169 16 3599 3583 0.0433169 0 - 0.36018 2024-04-22T07:41:54.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5170 16 3599 3583 0.0433085 0 - 0.36018 2024-04-22T07:41:55.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5171 16 3599 3583 0.0433002 0 - 0.36018 2024-04-22T07:41:56.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5172 16 3599 3583 0.0432918 0 - 0.36018 2024-04-22T07:41:57.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5173 16 3599 3583 0.0432834 0 - 0.36018 2024-04-22T07:41:58.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5174 16 3599 3583 0.0432751 0 - 0.36018 2024-04-22T07:41:59.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5175 16 3599 3583 0.0432667 0 - 0.36018 2024-04-22T07:42:00.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5176 16 3599 3583 0.0432583 0 - 0.36018 2024-04-22T07:42:01.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5177 16 3599 3583 0.04325 0 - 0.36018 2024-04-22T07:42:02.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5178 16 3599 3583 0.0432416 0 - 0.36018 2024-04-22T07:42:03.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5179 16 3599 3583 0.0432333 0 - 0.36018 2024-04-22T07:42:04.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:42:04.196941+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:42:04.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:42:04.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5180 16 3599 3583 0.0432249 0 - 0.36018 2024-04-22T07:42:05.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5181 16 3599 3583 0.0432166 0 - 0.36018 2024-04-22T07:42:06.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5182 16 3599 3583 0.0432083 0 - 0.36018 2024-04-22T07:42:07.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5183 16 3599 3583 0.0431999 0 - 0.36018 2024-04-22T07:42:08.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5184 16 3599 3583 0.0431916 0 - 0.36018 2024-04-22T07:42:09.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5185 16 3599 3583 0.0431833 0 - 0.36018 2024-04-22T07:42:10.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5186 16 3599 3583 0.0431749 0 - 0.36018 2024-04-22T07:42:11.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5187 16 3599 3583 0.0431666 0 - 0.36018 2024-04-22T07:42:12.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5188 16 3599 3583 0.0431583 0 - 0.36018 2024-04-22T07:42:13.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5189 16 3599 3583 0.04315 0 - 0.36018 2024-04-22T07:42:14.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5190 16 3599 3583 0.0431417 0 - 0.36018 2024-04-22T07:42:15.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5191 16 3599 3583 0.0431333 0 - 0.36018 2024-04-22T07:42:16.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5192 16 3599 3583 0.043125 0 - 0.36018 2024-04-22T07:42:17.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5193 16 3599 3583 0.0431167 0 - 0.36018 2024-04-22T07:42:18.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5194 16 3599 3583 0.0431084 0 - 0.36018 2024-04-22T07:42:19.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5195 16 3599 3583 0.0431001 0 - 0.36018 2024-04-22T07:42:20.197 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5196 16 3599 3583 0.0430918 0 - 0.36018 2024-04-22T07:42:21.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5197 16 3599 3583 0.0430835 0 - 0.36018 2024-04-22T07:42:22.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5198 16 3599 3583 0.0430753 0 - 0.36018 2024-04-22T07:42:23.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5199 16 3599 3583 0.043067 0 - 0.36018 2024-04-22T07:42:24.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-04-22T07:42:24.199888+0000 min lat: 0.0145691 max lat: 35.5965 avg lat: 0.36018 2024-04-22T07:42:24.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T07:42:24.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5200 16 3599 3583 0.0430587 0 - 0.36018 2024-04-22T07:42:25.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5201 16 3599 3583 0.0430504 0 - 0.36018 2024-04-22T07:42:26.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5202 16 3599 3583 0.0430421 0 - 0.36018 2024-04-22T07:42:27.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5203 16 3599 3583 0.0430339 0 - 0.36018 2024-04-22T07:42:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T07:42:27.429 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-22T07:42:27.452 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T07:42:27.452 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T07:42:27.452 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T07:42:27.452 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T07:42:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:27.453 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-04-22T07:42:27.453 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T07:42:27.453 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T07:42:27.453 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T07:42:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:27.454 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-22T07:42:27.454 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T07:42:27.454 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T07:42:27.454 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T07:42:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:27.455 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-04-22T07:42:27.455 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T07:42:27.455 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T07:42:27.455 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T07:42:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:27.455 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2024-04-22T07:42:27.456 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T07:42:27.456 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T07:42:27.456 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T07:42:27.456 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T07:42:27.456 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T07:42:27.456 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T07:42:27.456 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T07:42:27.457 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T07:42:27.457 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T07:42:27.457 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T07:42:27.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-22T07:42:27.498 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T07:42:27.498 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T07:42:27.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T07:42:27.609 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T07:42:27.664 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T07:42:27.664 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T07:42:27.736 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T07:42:27.736 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-22T07:42:27.793 INFO:tasks.ceph:Archiving mon data... 2024-04-22T07:42:27.794 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/data/mon.a.tgz 2024-04-22T07:42:27.794 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-04-22T07:42:27.811 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.CqkTgEIYfa 2024-04-22T07:42:27.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.CqkTgEIYfa 2024-04-22T07:42:27.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.CqkTgEIYfa 2024-04-22T07:42:28.018 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.CqkTgEIYfa is 238KB 2024-04-22T07:42:28.112 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.CqkTgEIYfa 2024-04-22T07:42:28.127 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T07:42:28.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T07:42:28.169 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T07:42:28.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5204 16 3599 3583 0.0430256 0 - 0.36018 2024-04-22T07:42:28.214 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T07:42:28.255 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T07:42:28.255 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T07:42:28.257 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/remote/smithi028/crash 2024-04-22T07:42:28.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T07:42:28.282 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/remote/smithi103/crash 2024-04-22T07:42:28.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T07:42:28.335 INFO:tasks.ceph:Compressing logs... 2024-04-22T07:42:28.336 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T07:42:28.340 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31249.log 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31492.log 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31339.log 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31249.log.gz 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31918.log 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31492.log.gz 2024-04-22T07:42:28.401 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28977.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31339.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30085.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31918.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29863.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28977.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29582.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30085.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31796.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29863.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31431.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29582.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29316.log 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31796.log.gz 2024-04-22T07:42:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31431.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30134.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29316.log.gz 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27759.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30134.log.gz 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31888.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27759.log.gz 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 91.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32167.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31735.log 2024-04-22T07:42:28.403 INFO:teuthology.orchestra.run.smithi028.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.31888.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30551.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32167.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31125.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.31735.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30551.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30551.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31948.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.31125.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29533.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28935.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31948.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28071.log 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29533.log.gz 2024-04-22T07:42:28.404 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.28935.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28179.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28935.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28409.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28071.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30995.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28179.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28239.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28409.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.30995.log: /var/log/ceph/ceph-client.admin.28683.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30995.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27801.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28239.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31765.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28683.log.gz 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28535.log 2024-04-22T07:42:28.405 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27801.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32103.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31765.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28575.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28535.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27903.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32103.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28893.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28575.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28515.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27903.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29764.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28893.log.gz 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32137.log 2024-04-22T07:42:28.406 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28515.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27675.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29764.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32137.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32137.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31035.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27675.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27082.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.31035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27717.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31035.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31401.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27082.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27717.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27717.log.gz 2024-04-22T07:42:28.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28199.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31401.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31185.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31065.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28199.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31858.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31185.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31065.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31065.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27161.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30827.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31858.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31155.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27161.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29103.log 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-04-22T07:42:28.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30701.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31155.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30743.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29103.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30230.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30701.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27346.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30743.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28367.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30230.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27346.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27346.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31309.log 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28367.log.gz 2024-04-22T07:42:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30868.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31309.log: 8.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31309.log.gz 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31279.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31095.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30868.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30868.log.gz 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28347.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31279.log.gz 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31615.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31095.log.gz 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31980.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28347.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28347.log.gz 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31584.log 2024-04-22T07:42:28.410 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31615.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32011.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.31980.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31553.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31584.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31584.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29061.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32011.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29019.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31553.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27900.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29061.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29061.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30183.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29019.log.gz 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31523.log 2024-04-22T07:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27900.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28744.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30183.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28068.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31523.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30659.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28744.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27317.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28068.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28068.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32227.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30659.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31371.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27317.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28238.log 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32227.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.32227.log.gz 2024-04-22T07:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31826.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31371.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27902.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28238.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29722.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31826.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31826.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27902.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28851.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29722.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29339.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29484.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29624.log 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28851.log.gz 2024-04-22T07:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.29339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30954.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.0.29339.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30909.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29624.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29624.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30954.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26021.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30954.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30452.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30909.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32257.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.tmp-client.admin.26021.log: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26021.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29673.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30452.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32197.log 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.32257.log.gz 2024-04-22T07:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28011.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29673.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29673.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32289.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32197.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28576.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28011.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27625.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32289.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32289.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32072.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28576.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30785.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27625.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29243.log 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32072.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32072.log.gz 2024-04-22T07:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.30785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31645.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30785.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29814.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29243.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31705.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31645.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.31645.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31462.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29814.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27901.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31705.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28743.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31462.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.31462.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28405.log 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27901.log.gz 2024-04-22T07:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32042.log 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28743.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27569.log 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28405.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28405.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29165.log 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32042.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29244.log 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27569.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28070.log 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29165.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29165.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.29244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28703.log 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29244.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28070.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28070.log.gz 2024-04-22T07:42:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.28703.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28703.log.gz 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27005.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27085.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.27005.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27005.log.gz 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27125.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.27085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27045.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27085.log.gz 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.27125.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27125.log.gz 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.27045.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-22T07:42:28.420 INFO:teuthology.orchestra.run.smithi103.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27045.log.gz 2024-04-22T07:42:28.446 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-mgr.x.log: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-22T07:42:28.508 INFO:teuthology.orchestra.run.smithi028.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-22T07:42:29.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5205 16 3599 3583 0.0430173 0 - 0.36018 2024-04-22T07:42:29.234 INFO:teuthology.orchestra.run.smithi028.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-22T07:42:29.326 INFO:teuthology.orchestra.run.smithi103.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-22T07:42:29.438 INFO:teuthology.orchestra.run.smithi103.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-22T07:42:29.440 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-22T07:42:29.440 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m1.045s 2024-04-22T07:42:29.440 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m1.932s 2024-04-22T07:42:29.440 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.081s 2024-04-22T07:42:29.475 INFO:teuthology.orchestra.run.smithi028.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-22T07:42:29.476 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-22T07:42:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m1.125s 2024-04-22T07:42:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m2.081s 2024-04-22T07:42:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.119s 2024-04-22T07:42:29.477 INFO:tasks.ceph:Archiving logs... 2024-04-22T07:42:29.477 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/remote/smithi028/log 2024-04-22T07:42:29.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:42:29.997 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/remote/smithi103/log 2024-04-22T07:42:29.998 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:42:30.226 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5206 16 3599 3583 0.0430091 0 - 0.36018 2024-04-22T07:42:30.356 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-22T07:42:30.356 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T07:42:30.366 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds 2024-04-22T07:42:30.367 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T07:42:30.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:42:30.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:42:30.463 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T07:42:30.472 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T07:42:30.472 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:42:30.475 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:42:30.488 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:42:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:42:30.492 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:42:30.492 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-22T07:42:30.492 INFO:teuthology.orchestra.run.smithi103.stdout:^- hv01.front.sepia.ceph.com 2 7 377 66 +1318us[+1318us] +/- 619ms 2024-04-22T07:42:30.492 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 8 377 201 +82us[ +52us] +/- 58ms 2024-04-22T07:42:30.493 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 71 -88us[ -88us] +/- 62ms 2024-04-22T07:42:30.493 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:42:30.494 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:42:30.494 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-22T07:42:30.494 INFO:teuthology.orchestra.run.smithi028.stdout:^- hv01.front.sepia.ceph.com 2 7 377 64 +1307us[+1307us] +/- 619ms 2024-04-22T07:42:30.494 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 8 377 134 +87us[ +51us] +/- 59ms 2024-04-22T07:42:30.494 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -145us[ -145us] +/- 63ms 2024-04-22T07:42:30.494 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:42:30.495 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T07:42:30.505 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T07:42:30.506 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T07:42:30.515 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:42:30.605 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:42:30.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:42:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:30.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:42:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:30.718 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T07:42:30.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T07:42:30.737 INFO:teuthology.task.internal:Duration was 5658.225421 seconds 2024-04-22T07:42:30.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T07:42:30.757 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T07:42:30.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:42:30.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:42:30.795 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:42:30.799 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:42:31.004 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T07:42:31.004 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-22T07:42:31.005 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:42:31.058 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-22T07:42:31.058 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:42:31.085 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T07:42:31.086 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:42:31.100 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:42:31.150 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T07:42:31.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:42:31.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:42:31.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5207 16 3599 3583 0.0430008 0 - 0.36018 2024-04-22T07:42:31.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T07:42:31.298 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T07:42:31.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:42:31.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:42:31.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T07:42:31.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:42:31.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:42:31.405 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-22T07:42:31.428 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-22T07:42:31.473 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:42:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:31.488 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:42:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:42:31.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T07:42:31.538 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T07:42:31.539 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/remote/smithi028 2024-04-22T07:42:31.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:42:31.573 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179/remote/smithi103 2024-04-22T07:42:31.573 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:42:31.604 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T07:42:31.605 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:42:31.610 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:42:31.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T07:42:31.666 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T07:42:31.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T07:42:31.676 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T07:42:31.676 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:42:31.678 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:42:31.691 INFO:teuthology.orchestra.run.smithi028.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:42 /home/ubuntu/cephtest 2024-04-22T07:42:31.708 INFO:teuthology.orchestra.run.smithi103.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:42 /home/ubuntu/cephtest 2024-04-22T07:42:31.709 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T07:42:31.721 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T07:42:31.736 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (801) after waiting for 4800 seconds 2024-04-22T07:42:31.736 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T07:42:31.771 DEBUG:teuthology.nuke:shortname: smithi028 2024-04-22T07:42:31.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:42:31.812 DEBUG:teuthology.nuke:shortname: smithi103 2024-04-22T07:42:31.812 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:42:31.825 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:02:01.188758', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCArwe8JY69vRPI1gOppoAfuoqH5tvZqOUAgJncBEpBp/SABpASB1UDcRygqcH3J18xC9OFcHZCwRO4oq2Y24ak='} 2024-04-22T07:42:31.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:02:01.187508', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGX9O9cuJkHoW43qGAdaoeyYRTcsLwDfmJ7iTXZTE/RP482xLx50nETZMnXxLAlwMqOuk7JBZyCUl9Qk+TYaugA='} 2024-04-22T07:42:31.920 INFO:teuthology.orchestra.console.smithi028:Power off 2024-04-22T07:42:31.920 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:42:31.966 INFO:teuthology.orchestra.console.smithi103:Power off 2024-04-22T07:42:31.966 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:42:31.974 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-04-22T07:42:31.974 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:32.013 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-04-22T07:42:32.014 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:32.023 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-22T07:42:32.063 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-22T07:42:32.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5208 16 3599 3583 0.0429926 0 - 0.36018 2024-04-22T07:42:33.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5209 16 3599 3583 0.0429843 0 - 0.36018 2024-04-22T07:42:34.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5210 16 3599 3583 0.042976 0 - 0.36018 2024-04-22T07:42:35.199 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5211 16 3599 3583 0.0429678 0 - 0.36018 2024-04-22T07:42:36.024 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:36.064 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:36.133 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-22T07:42:36.174 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-22T07:42:40.138 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:40.174 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:40.247 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-04-22T07:42:40.288 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-22T07:42:40.349 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-04-22T07:42:44.290 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:42:44.400 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-04-22T07:42:44.501 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-04-22T07:42:44.605 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} duration: 5658.225420713425 failure_reason: reached maximum tries (801) after waiting for 4800 seconds flavor: crimson owner: scheduled_yingxin@teuthology status: fail success: false 2024-04-22T07:42:44.605 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T07:42:44.714 INFO:teuthology.run:FAIL