2024-04-16T14:45:52.178 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T14:45:52.178 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391 --owner scheduled_matan@teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} --verbose --name matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi -- /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/orig.config.yaml 2024-04-16T14:45:52.223 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T14:45:52.305 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391 branch: main description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7658391' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 3f7db60e5ba4c4d053b594b7ec58703d2833c55b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 3f7db60e5ba4c4d053b594b7ec58703d2833c55b selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-thrash-osd env: CRIMSON_COMPAT: '1' sha1: 372751789be1725ed06fd077d029295c8cab4c35 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 3903 sha1: 3f7db60e5ba4c4d053b594b7ec58703d2833c55b sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-thrash-osd suite_path: /home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa suite_relpath: qa suite_repo: https://github.com/Matan-B/ceph.git suite_sha1: 372751789be1725ed06fd077d029295c8cab4c35 targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNXmzu0aqrX/brCJad7hOOc5xrCmRP7WJ6BEU7E69d9quoupN6W7mtUvqmVhcN3YHewmLUVRlhtjINfAKQI77AI= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjG692fe51fvXOSbVE4fA2zMap3ZdmEsjONTdJZdF9fG3q77p0KwYF+U68bWtBouhDOJwkcwHw+f4VGRs5dCew= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_14:35:17 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T14:45:52.306 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa; will attempt to use it 2024-04-16T14:45:52.306 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks 2024-04-16T14:45:52.307 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T14:45:52.308 INFO:teuthology.task.internal:Checking packages... 2024-04-16T14:45:52.332 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '3f7db60e5ba4c4d053b594b7ec58703d2833c55b' 2024-04-16T14:45:52.333 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T14:45:52.333 INFO:teuthology.packaging:ref: None 2024-04-16T14:45:52.333 INFO:teuthology.packaging:tag: None 2024-04-16T14:45:52.333 INFO:teuthology.packaging:branch: main 2024-04-16T14:45:52.333 INFO:teuthology.packaging:sha1: 3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T14:45:52.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-04-16T14:45:52.558 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3100.g3f7db60e 2024-04-16T14:45:52.559 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T14:45:52.571 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T14:45:52.571 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T14:45:52.586 INFO:teuthology.task.internal:Saving configuration 2024-04-16T14:45:52.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T14:45:52.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T14:45:52.644 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 14:40:01.376396', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNXmzu0aqrX/brCJad7hOOc5xrCmRP7WJ6BEU7E69d9quoupN6W7mtUvqmVhcN3YHewmLUVRlhtjINfAKQI77AI='} 2024-04-16T14:45:52.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 14:40:01.377640', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjG692fe51fvXOSbVE4fA2zMap3ZdmEsjONTdJZdF9fG3q77p0KwYF+U68bWtBouhDOJwkcwHw+f4VGRs5dCew='} 2024-04-16T14:45:52.667 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T14:45:52.678 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-16T14:45:52.679 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-16T14:45:52.679 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T14:45:52.757 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff9bd943e50>, signals=[15]) 2024-04-16T14:45:52.757 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T14:45:52.769 INFO:teuthology.task.internal:Opening connections... 2024-04-16T14:45:52.769 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T14:45:52.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:45:52.864 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T14:45:52.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:45:52.953 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T14:45:52.966 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-04-16T14:45:52.984 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-04-16T14:45:52.984 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-04-16T14:45:53.040 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T14:45:53.041 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T14:45:53.042 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-04-16T14:45:53.063 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-16T14:45:53.081 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-16T14:45:53.081 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-16T14:45:53.138 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T14:45:53.139 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-16T14:45:53.139 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T14:45:53.139 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T14:45:53.139 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T14:45:53.139 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-16T14:45:53.179 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T14:45:53.189 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T14:45:53.196 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T14:45:53.196 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-04-16T14:45:53.199 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-16T14:45:53.215 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T14:45:53.231 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T14:45:53.231 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-04-16T14:45:53.255 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-16T14:45:53.363 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T14:45:53.418 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T14:45:53.437 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T14:45:53.437 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T14:45:53.437 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T14:45:53.438 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T14:45:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T14:45:53.454 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-04-16T14:45:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T14:45:53.455 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-04-16T14:45:53.511 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2024-04-16T14:45:53.511 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.14.0-361.el9.x86_64 2024-04-16T14:45:53.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2024-04-16T14:45:53.513 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-04-16T14:45:53.513 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-04-16T14:45:53.513 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-04-16T14:45:55.802 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.7 MB 00:00 2024-04-16T14:45:57.335 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-04-16T14:45:57.805 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 2.5 MB/s | 6.7 MB 00:02 2024-04-16T14:45:59.429 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-04-16T14:45:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-16T14:46:00.353 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-16T14:46:02.379 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-16T14:46:06.326 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 19 MB 00:05 2024-04-16T14:46:07.872 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.2 MB 00:01 2024-04-16T14:46:09.448 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-16T14:46:12.035 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 21 MB 00:02 2024-04-16T14:46:12.227 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.2 MB 00:02 2024-04-16T14:46:13.719 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-16T14:46:16.806 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-04-16T14:46:17.098 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-16T14:46:19.472 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T14:46:19.472 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T14:46:19.564 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:46:19.574 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:46:19.574 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2024-04-16T14:46:19.575 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:46:19.576 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 77 M 2024-04-16T14:46:19.576 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 123 M 2024-04-16T14:46:19.576 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:46:19.792 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.3 MB/s | 277 kB 00:00 2024-04-16T14:46:20.758 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-04-16T14:46:21.870 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-16T14:46:22.667 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 11 MB/s | 34 MB 00:03 2024-04-16T14:46:23.175 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 8.2 MB/s | 28 MB 00:03 2024-04-16T14:46:23.178 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:46:23.179 INFO:teuthology.orchestra.run.smithi097.stdout:Total 21 MB/s | 77 MB 00:03 2024-04-16T14:46:23.179 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:46:23.531 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:46:23.531 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:46:24.266 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T14:46:24.266 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T14:46:24.351 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:46:24.351 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:46:24.356 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:46:24.372 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:46:24.372 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:46:24.372 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:46:24.372 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-04-16T14:46:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:46:24.374 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 77 M 2024-04-16T14:46:24.374 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-04-16T14:46:24.374 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:46:24.538 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-16T14:46:25.096 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-04-16T14:46:25.663 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-04-16T14:46:26.175 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-16T14:46:26.175 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:46:26.175 INFO:teuthology.orchestra.run.smithi186.stdout:Total 43 MB/s | 77 MB 00:01 2024-04-16T14:46:26.175 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:46:26.563 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:46:26.563 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:46:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:46:27.259 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T14:46:27.350 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T14:46:27.376 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:46:27.376 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:46:29.690 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:46:30.092 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T14:46:30.229 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T14:46:30.356 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T14:46:30.447 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T14:46:33.202 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T14:46:33.333 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T14:46:34.911 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T14:46:34.945 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:46:38.029 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T14:46:38.156 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:46:39.697 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:46:42.884 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:08.896 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:08.896 INFO:teuthology.orchestra.run.smithi097.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T14:47:08.896 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:47:08.939 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T14:47:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T14:47:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T14:47:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:47:09.703 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:47:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T14:47:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T14:47:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T14:47:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T14:47:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:47:09.704 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:47:10.026 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T14:47:10.126 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T14:47:10.127 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T14:47:10.127 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T14:47:10.127 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T14:47:10.127 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T14:47:10.127 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T14:47:10.127 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T14:47:10.127 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T14:47:10.207 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T14:47:10.208 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T14:47:10.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T14:47:10.242 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T14:47:10.243 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-04-16T14:47:10.244 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:47:10.244 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T14:47:10.244 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T14:47:10.245 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 5.14.0-438.el9.x86_64 2024-04-16T14:47:10.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T14:47:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-04-16T14:47:12.248 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:12.248 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T14:47:12.249 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:47:12.280 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T14:47:12.292 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:12.318 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-04-16T14:47:12.321 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-04-16T14:47:12.335 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.o3EJtdxxjx 2024-04-16T14:47:12.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.o3EJtdxxjx 2024-04-16T14:47:12.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.o3EJtdxxjx 2024-04-16T14:47:12.542 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.o3EJtdxxjx is 6KB 2024-04-16T14:47:12.590 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.o3EJtdxxjx 2024-04-16T14:47:12.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T14:47:12.670 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T14:47:12.670 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T14:47:12.670 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T14:47:12.670 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T14:47:12.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T14:47:12.712 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:12.712 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T14:47:12.712 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T14:47:12.712 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T14:47:12.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T14:47:13.107 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T14:47:13.108 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:47:13.108 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:47:13.450 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T14:47:13.562 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T14:47:13.562 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T14:47:13.563 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T14:47:13.563 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T14:47:13.563 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T14:47:13.563 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-16T14:47:13.563 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T14:47:13.563 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T14:47:13.643 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T14:47:13.643 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T14:47:13.643 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-04-16T14:47:13.677 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-04-16T14:47:13.677 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-04-16T14:47:13.677 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-04-16T14:47:13.678 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:47:13.679 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T14:47:13.680 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T14:47:13.680 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-438.el9.x86_64 2024-04-16T14:47:13.680 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T14:47:14.277 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-04-16T14:47:15.653 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T14:47:15.690 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-04-16T14:47:15.692 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-16T14:47:15.707 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.K3op5HVDxe 2024-04-16T14:47:15.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K3op5HVDxe 2024-04-16T14:47:15.777 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.K3op5HVDxe 2024-04-16T14:47:15.918 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.K3op5HVDxe is 6KB 2024-04-16T14:47:15.966 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.K3op5HVDxe 2024-04-16T14:47:15.980 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T14:47:16.046 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T14:47:16.046 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T14:47:16.046 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T14:47:16.046 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T14:47:16.047 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T14:47:16.204 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-04-16T14:47:42.853 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T14:47:42.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-16T14:47:42.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:47:46.210 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-16T14:47:46.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-16T14:47:46.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:48:20.767 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-16T14:48:29.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-16T14:48:29.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:48:42.871 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T14:48:51.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-16T14:48:51.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:48:52.269 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-16T14:48:52.668 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-16T14:48:52.668 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T14:48:52.669 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-04-16T14:48:52.683 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-438.el9.x86_64 2024-04-16T14:48:52.684 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T14:48:52.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T14:48:52.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T14:48:53.684 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T14:48:53.685 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T14:48:53.761 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-04-16T14:48:53.814 DEBUG:teuthology.parallel:result is None 2024-04-16T14:49:29.778 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T14:49:41.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-16T14:49:41.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:49:42.260 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-16T14:49:42.623 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-16T14:49:42.623 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-438.el9.x86_64"... 2024-04-16T14:49:42.623 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-04-16T14:49:42.638 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-438.el9.x86_64 2024-04-16T14:49:42.639 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T14:49:42.639 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T14:49:42.639 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T14:49:43.640 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-16T14:49:43.640 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T14:49:43.712 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-04-16T14:49:43.747 DEBUG:teuthology.parallel:result is None 2024-04-16T14:49:43.747 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T14:49:43.755 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T14:49:43.756 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T14:49:43.759 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T14:49:43.776 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T14:49:43.846 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T14:49:43.853 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T14:49:43.853 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T14:49:43.856 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T14:49:43.887 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T14:49:43.894 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T14:49:43.894 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T14:49:43.921 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T14:49:43.963 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T14:49:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T14:49:43.977 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T14:49:43.980 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T14:49:43.981 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T14:49:43.989 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T14:49:43.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T14:49:43.992 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T14:49:44.053 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T14:49:44.062 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T14:49:44.062 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T14:49:44.064 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T14:49:44.110 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T14:49:44.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T14:49:44.218 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T14:49:44.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T14:49:44.345 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:49:44.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T14:49:44.413 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T14:49:44.456 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T14:49:44.516 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T14:49:44.595 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T14:49:44.651 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:49:44.651 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T14:49:44.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-04-16T14:49:44.724 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-16T14:49:44.765 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T14:49:44.816 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T14:49:45.140 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T14:49:45.149 INFO:teuthology.task.internal:Starting timer... 2024-04-16T14:49:45.149 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T14:49:45.159 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T14:49:45.168 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T14:49:45.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-04-16T14:49:45.258 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-04-16T14:49:45.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-16T14:49:45.307 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-16T14:49:45.309 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T14:49:45.309 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-04-16T14:49:45.333 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-04-16T14:49:45.333 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-16T14:49:45.359 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-16T14:49:45.359 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-16T14:49:45.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T14:49:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T14:49:45.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T14:49:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T14:49:45.432 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T14:49:45.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-04-16T14:49:45.477 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-16T14:49:45.520 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T14:49:45.530 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T14:49:45.549 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T14:49:45.551 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T14:54:10.695 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-16T14:54:10.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-16T14:54:10.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:54:10.790 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-16T14:54:10.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-16T14:54:10.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-16T14:54:10.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T14:54:10.959 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-16T14:54:11.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-16T14:54:11.042 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T14:54:11.053 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T14:54:11.053 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T14:54:11.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T14:54:11.056 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T14:54:11.057 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T14:54:11.088 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T14:54:11.106 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T14:54:11.115 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T14:54:11.133 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T14:54:11.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-04-16T14:54:11.152 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-04-16T14:54:11.164 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-16T14:54:11.169 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T14:54:11.178 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-16T14:54:11.186 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T14:54:11.196 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T14:54:11.213 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T14:54:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-04-16T14:54:11.247 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T14:54:11.247 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-04-16T14:54:11.247 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.247 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.247 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.247 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-16T14:54:11.271 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T14:54:11.271 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-16T14:54:11.271 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.271 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.271 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.271 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T14:54:11.272 INFO:teuthology.run_tasks:Running task install... 2024-04-16T14:54:11.280 DEBUG:teuthology.task.install:project ceph 2024-04-16T14:54:11.280 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'}} 2024-04-16T14:54:11.280 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'} 2024-04-16T14:54:11.280 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T14:54:11.285 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T14:54:11.286 INFO:teuthology.task.install:extra packages: [] 2024-04-16T14:54:11.286 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b', 'tag': None, 'wait_for_package': False} 2024-04-16T14:54:11.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T14:54:11.288 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b', 'tag': None, 'wait_for_package': False} 2024-04-16T14:54:11.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T14:54:11.432 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/ 2024-04-16T14:54:11.432 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3100.g3f7db60e 2024-04-16T14:54:11.438 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/ 2024-04-16T14:54:11.438 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3100.g3f7db60e 2024-04-16T14:54:11.814 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T14:54:11.815 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:54:11.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T14:54:11.847 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T14:54:11.847 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T14:54:11.917 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T14:54:12.000 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T14:54:12.071 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-16T14:54:12.071 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-16T14:54:12.319 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-04-16T14:54:12.347 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw 2024-04-16T14:54:13.631 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 89 kB/s | 87 kB 00:00 2024-04-16T14:54:14.457 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 15 kB/s | 12 kB 00:00 2024-04-16T14:54:14.801 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T14:54:14.802 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:54:14.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T14:54:14.836 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T14:54:14.837 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3f7db60e5ba4c4d053b594b7ec58703d2833c55b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T14:54:14.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T14:54:14.996 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T14:54:15.066 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-04-16T14:54:15.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-04-16T14:54:15.314 INFO:teuthology.orchestra.run.smithi097.stdout:74 files removed 2024-04-16T14:54:15.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2024-04-16T14:54:15.408 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 2.2 kB/s | 2.0 kB 00:00 2024-04-16T14:54:16.043 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T14:54:17.062 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 61 kB/s | 87 kB 00:01 2024-04-16T14:54:17.652 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-04-16T14:54:17.846 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 16 kB/s | 12 kB 00:00 2024-04-16T14:54:18.797 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 2.3 kB/s | 2.0 kB 00:00 2024-04-16T14:54:19.574 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 9.1 MB/s | 6.7 MB 00:00 2024-04-16T14:54:21.058 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-04-16T14:54:21.192 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-16T14:54:21.840 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-04-16T14:54:23.791 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-16T14:54:27.276 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 19 MB 00:04 2024-04-16T14:54:29.792 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.2 MB 00:02 2024-04-16T14:54:31.451 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-04-16T14:54:33.044 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.2 MB 00:02 2024-04-16T14:54:33.322 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T14:54:34.736 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-04-16T14:54:36.721 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-16T14:54:38.453 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 9.7 kB/s | 1.7 kB 00:00 2024-04-16T14:54:41.198 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:54:41.208 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:54:41.208 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 10 M 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 3.4 M 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 3.7 M 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:54:41.209 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 5.9 M 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 19 M 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 25 k 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 796 k 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T14:54:41.210 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 548 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 5.3 M 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T14:54:41.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 45 k 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 135 k 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 159 k 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 316 k 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 292 k 2024-04-16T14:54:41.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 99 k 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T14:54:41.213 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:54:41.214 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:54:41.214 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:54:41.214 INFO:teuthology.orchestra.run.smithi097.stdout:Install 30 Packages 2024-04-16T14:54:41.214 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2024-04-16T14:54:41.214 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:54:41.215 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 59 M 2024-04-16T14:54:41.215 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:54:41.771 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-16T14:54:44.525 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 10 M 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 3.4 M 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 3.7 M 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options 2024-04-16T14:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 5.9 M 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 19 M 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 25 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 796 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 548 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 5.3 M 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T14:54:44.543 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 45 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 135 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 159 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 316 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 292 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 99 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T14:54:44.544 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout:Install 30 Packages 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 2 Packages 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 59 M 2024-04-16T14:54:44.545 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:54:47.064 INFO:teuthology.orchestra.run.smithi186.stdout:(1/32): ceph-base-19.0.0-3100.g3f7db60e.el9.x86 2.7 MB/s | 5.9 MB 00:02 2024-04-16T14:54:47.164 INFO:teuthology.orchestra.run.smithi186.stdout:(2/32): ceph-selinux-19.0.0-3100.g3f7db60e.el9. 252 kB/s | 25 kB 00:00 2024-04-16T14:54:47.372 INFO:teuthology.orchestra.run.smithi186.stdout:(3/32): libcephfs2-19.0.0-3100.g3f7db60e.el9.x8 3.7 MB/s | 796 kB 00:00 2024-04-16T14:54:47.565 INFO:teuthology.orchestra.run.smithi186.stdout:(4/32): libradosstriper1-19.0.0-3100.g3f7db60e. 2.8 MB/s | 548 kB 00:00 2024-04-16T14:54:48.623 INFO:teuthology.orchestra.run.smithi186.stdout:(5/32): librgw2-19.0.0-3100.g3f7db60e.el9.x86_6 5.0 MB/s | 5.3 MB 00:01 2024-04-16T14:54:48.731 INFO:teuthology.orchestra.run.smithi186.stdout:(6/32): python3-ceph-argparse-19.0.0-3100.g3f7d 417 kB/s | 45 kB 00:00 2024-04-16T14:54:49.557 INFO:teuthology.orchestra.run.smithi186.stdout:(7/32): python3-ceph-common-19.0.0-3100.g3f7db6 163 kB/s | 135 kB 00:00 2024-04-16T14:54:49.682 INFO:teuthology.orchestra.run.smithi186.stdout:(8/32): python3-cephfs-19.0.0-3100.g3f7db60e.el 1.2 MB/s | 159 kB 00:00 2024-04-16T14:54:50.449 INFO:teuthology.orchestra.run.smithi186.stdout:(9/32): python3-rados-19.0.0-3100.g3f7db60e.el9 412 kB/s | 316 kB 00:00 2024-04-16T14:54:51.799 INFO:teuthology.orchestra.run.smithi186.stdout:(10/32): python3-rbd-19.0.0-3100.g3f7db60e.el9. 217 kB/s | 292 kB 00:01 2024-04-16T14:54:52.191 INFO:teuthology.orchestra.run.smithi186.stdout:(11/32): python3-rgw-19.0.0-3100.g3f7db60e.el9. 253 kB/s | 99 kB 00:00 2024-04-16T14:54:52.325 INFO:teuthology.orchestra.run.smithi186.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 795 kB/s | 106 kB 00:00 2024-04-16T14:54:52.366 INFO:teuthology.orchestra.run.smithi186.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-04-16T14:54:52.400 INFO:teuthology.orchestra.run.smithi186.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T14:54:52.450 INFO:teuthology.orchestra.run.smithi186.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-04-16T14:54:52.492 INFO:teuthology.orchestra.run.smithi186.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-16T14:54:52.526 INFO:teuthology.orchestra.run.smithi186.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-04-16T14:54:52.560 INFO:teuthology.orchestra.run.smithi186.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2024-04-16T14:54:52.593 INFO:teuthology.orchestra.run.smithi186.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-04-16T14:54:52.627 INFO:teuthology.orchestra.run.smithi186.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-04-16T14:54:53.144 INFO:teuthology.orchestra.run.smithi186.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 598 kB/s | 309 kB 00:00 2024-04-16T14:54:53.319 INFO:teuthology.orchestra.run.smithi186.stdout:(22/32): ceph-radosgw-19.0.0-3100.g3f7db60e.el9 1.2 MB/s | 10 MB 00:08 2024-04-16T14:54:53.594 INFO:teuthology.orchestra.run.smithi186.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-04-16T14:54:53.830 INFO:teuthology.orchestra.run.smithi186.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 208 kB/s | 49 kB 00:00 2024-04-16T14:54:54.045 INFO:teuthology.orchestra.run.smithi186.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 311 kB/s | 67 kB 00:00 2024-04-16T14:54:55.512 INFO:teuthology.orchestra.run.smithi186.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 571 kB/s | 838 kB 00:01 2024-04-16T14:54:55.519 INFO:teuthology.orchestra.run.smithi097.stdout:(1/32): ceph-base-19.0.0-3100.g3f7db60e.el9.x86 426 kB/s | 5.9 MB 00:14 2024-04-16T14:54:55.619 INFO:teuthology.orchestra.run.smithi097.stdout:(2/32): ceph-selinux-19.0.0-3100.g3f7db60e.el9. 252 kB/s | 25 kB 00:00 2024-04-16T14:54:55.787 INFO:teuthology.orchestra.run.smithi186.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 697 kB/s | 192 kB 00:00 2024-04-16T14:54:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:(3/32): libcephfs2-19.0.0-3100.g3f7db60e.el9.x8 796 kB/s | 796 kB 00:00 2024-04-16T14:54:57.062 INFO:teuthology.orchestra.run.smithi186.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:03 2024-04-16T14:54:57.379 INFO:teuthology.orchestra.run.smithi186.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 1.0 MB/s | 1.6 MB 00:01 2024-04-16T14:54:57.978 INFO:teuthology.orchestra.run.smithi097.stdout:(4/32): libradosstriper1-19.0.0-3100.g3f7db60e. 404 kB/s | 548 kB 00:01 2024-04-16T14:54:59.772 INFO:teuthology.orchestra.run.smithi186.stdout:(30/32): librados2-19.0.0-3100.g3f7db60e.el9.x8 1.3 MB/s | 3.4 MB 00:02 2024-04-16T14:55:00.587 INFO:teuthology.orchestra.run.smithi186.stdout:(31/32): librbd1-19.0.0-3100.g3f7db60e.el9.x86_ 1.1 MB/s | 3.7 MB 00:03 2024-04-16T14:55:03.128 INFO:teuthology.orchestra.run.smithi097.stdout:(5/32): librgw2-19.0.0-3100.g3f7db60e.el9.x86_6 1.0 MB/s | 5.3 MB 00:05 2024-04-16T14:55:03.228 INFO:teuthology.orchestra.run.smithi097.stdout:(6/32): python3-ceph-argparse-19.0.0-3100.g3f7d 451 kB/s | 45 kB 00:00 2024-04-16T14:55:03.412 INFO:teuthology.orchestra.run.smithi097.stdout:(7/32): python3-ceph-common-19.0.0-3100.g3f7db6 735 kB/s | 135 kB 00:00 2024-04-16T14:55:03.929 INFO:teuthology.orchestra.run.smithi097.stdout:(8/32): python3-cephfs-19.0.0-3100.g3f7db60e.el 309 kB/s | 159 kB 00:00 2024-04-16T14:55:04.213 INFO:teuthology.orchestra.run.smithi097.stdout:(9/32): ceph-common-19.0.0-3100.g3f7db60e.el9.x 869 kB/s | 19 MB 00:22 2024-04-16T14:55:04.396 INFO:teuthology.orchestra.run.smithi097.stdout:(10/32): python3-rados-19.0.0-3100.g3f7db60e.el 676 kB/s | 316 kB 00:00 2024-04-16T14:55:04.585 INFO:teuthology.orchestra.run.smithi097.stdout:(11/32): python3-rgw-19.0.0-3100.g3f7db60e.el9. 527 kB/s | 99 kB 00:00 2024-04-16T14:55:04.705 INFO:teuthology.orchestra.run.smithi097.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 885 kB/s | 106 kB 00:00 2024-04-16T14:55:04.747 INFO:teuthology.orchestra.run.smithi097.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-04-16T14:55:04.781 INFO:teuthology.orchestra.run.smithi097.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T14:55:04.806 INFO:teuthology.orchestra.run.smithi097.stdout:(15/32): python3-rbd-19.0.0-3100.g3f7db60e.el9. 493 kB/s | 292 kB 00:00 2024-04-16T14:55:04.834 INFO:teuthology.orchestra.run.smithi097.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-04-16T14:55:04.856 INFO:teuthology.orchestra.run.smithi097.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-04-16T14:55:04.882 INFO:teuthology.orchestra.run.smithi097.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-04-16T14:55:04.907 INFO:teuthology.orchestra.run.smithi097.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-16T14:55:04.932 INFO:teuthology.orchestra.run.smithi097.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 699 kB/s | 35 kB 00:00 2024-04-16T14:55:04.957 INFO:teuthology.orchestra.run.smithi097.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-04-16T14:55:05.408 INFO:teuthology.orchestra.run.smithi097.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 650 kB/s | 309 kB 00:00 2024-04-16T14:55:05.483 INFO:teuthology.orchestra.run.smithi097.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-04-16T14:55:05.566 INFO:teuthology.orchestra.run.smithi097.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 585 kB/s | 49 kB 00:00 2024-04-16T14:55:05.650 INFO:teuthology.orchestra.run.smithi097.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-04-16T14:55:05.726 INFO:teuthology.orchestra.run.smithi097.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-04-16T14:55:05.801 INFO:teuthology.orchestra.run.smithi097.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-16T14:55:05.851 INFO:teuthology.orchestra.run.smithi097.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-04-16T14:55:05.934 INFO:teuthology.orchestra.run.smithi097.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-16T14:55:06.185 INFO:teuthology.orchestra.run.smithi097.stdout:(30/32): ceph-radosgw-19.0.0-3100.g3f7db60e.el9 417 kB/s | 10 MB 00:24 2024-04-16T14:55:09.260 INFO:teuthology.orchestra.run.smithi097.stdout:(31/32): librados2-19.0.0-3100.g3f7db60e.el9.x8 1.0 MB/s | 3.4 MB 00:03 2024-04-16T14:55:11.844 INFO:teuthology.orchestra.run.smithi097.stdout:(32/32): librbd1-19.0.0-3100.g3f7db60e.el9.x86_ 638 kB/s | 3.7 MB 00:05 2024-04-16T14:55:11.849 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:55:11.849 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.9 MB/s | 59 MB 00:30 2024-04-16T14:55:11.849 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:55:11.887 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:55:11.887 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:55:12.361 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:55:12.361 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:55:13.829 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:55:14.324 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T14:55:14.362 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/34 2024-04-16T14:55:14.445 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/34 2024-04-16T14:55:14.534 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 3/34 2024-04-16T14:55:14.564 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/34 2024-04-16T14:55:14.610 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/34 2024-04-16T14:55:14.650 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T14:55:14.676 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T14:55:14.702 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T14:55:14.726 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T14:55:14.741 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T14:55:14.934 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T14:55:14.965 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T14:55:15.320 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T14:55:15.353 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/34 2024-04-16T14:55:15.412 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/34 2024-04-16T14:55:15.446 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-3100.g3f7db60e.el9. 12/34 2024-04-16T14:55:15.501 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 13/34 2024-04-16T14:55:15.526 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 14/34 2024-04-16T14:55:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T14:55:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_6 16/34 2024-04-16T14:55:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_6 16/34 2024-04-16T14:55:15.755 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T14:55:15.862 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T14:55:15.897 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T14:55:15.938 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T14:55:15.970 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T14:55:16.016 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T14:55:16.047 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T14:55:16.088 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T14:55:16.649 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T14:55:16.670 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 26/34 2024-04-16T14:55:16.742 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 26/34 2024-04-16T14:55:16.809 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 27/34 2024-04-16T14:55:16.839 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-3100.g3f7db60e.el9.x8 28/34 2024-04-16T14:55:19.310 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 29/34 2024-04-16T14:55:19.352 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 29/34 2024-04-16T14:55:20.017 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 29/34 2024-04-16T14:55:20.058 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 30/34 2024-04-16T14:55:20.141 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 30/34 2024-04-16T14:55:20.141 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T14:55:20.141 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T14:55:20.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:20.173 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 31/34 2024-04-16T14:55:21.620 INFO:teuthology.orchestra.run.smithi186.stdout:(32/32): ceph-common-19.0.0-3100.g3f7db60e.el9. 542 kB/s | 19 MB 00:36 2024-04-16T14:55:21.622 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:55:21.623 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.6 MB/s | 59 MB 00:37 2024-04-16T14:55:21.623 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:55:21.659 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:55:21.659 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:55:23.623 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:55:24.115 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T14:55:24.155 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/34 2024-04-16T14:55:24.245 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/34 2024-04-16T14:55:24.342 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 3/34 2024-04-16T14:55:24.373 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/34 2024-04-16T14:55:24.436 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/34 2024-04-16T14:55:24.483 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T14:55:24.520 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T14:55:24.545 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T14:55:24.567 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T14:55:24.583 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T14:55:24.728 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T14:55:24.758 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T14:55:25.108 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T14:55:25.129 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/34 2024-04-16T14:55:25.196 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/34 2024-04-16T14:55:25.239 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3100.g3f7db60e.el9. 12/34 2024-04-16T14:55:25.293 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 13/34 2024-04-16T14:55:25.318 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 14/34 2024-04-16T14:55:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T14:55:25.424 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_6 16/34 2024-04-16T14:55:25.511 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_6 16/34 2024-04-16T14:55:25.539 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T14:55:25.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T14:55:25.674 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T14:55:25.755 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T14:55:25.784 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T14:55:25.815 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T14:55:25.853 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T14:55:26.425 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T14:55:26.455 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 26/34 2024-04-16T14:55:26.526 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 26/34 2024-04-16T14:55:26.594 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 27/34 2024-04-16T14:55:26.624 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3100.g3f7db60e.el9.x8 28/34 2024-04-16T14:55:29.057 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 29/34 2024-04-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 29/34 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 31/34 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-04-16T14:55:29.608 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:29.775 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 29/34 2024-04-16T14:55:29.807 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 30/34 2024-04-16T14:55:29.879 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 30/34 2024-04-16T14:55:29.879 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T14:55:29.879 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T14:55:29.879 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:29.916 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 31/34 2024-04-16T14:55:30.669 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/34 2024-04-16T14:55:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/34 2024-04-16T14:55:30.699 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:55:30.699 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T14:55:30.699 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T14:55:30.700 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T14:55:30.700 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:30.701 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T14:55:30.787 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T14:55:30.788 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 3/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 5/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_6 6/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 7/34 2024-04-16T14:55:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3100.g3f7db60e.el9. 8/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-3100.g3f7db60e.el9.x8 9/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 10/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 12/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 13/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T14:55:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 31/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T14:55:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 33/34 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.166 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T14:55:33.167 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:33.168 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:55:33.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2024-04-16T14:55:34.123 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:56 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:55:34.286 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 84 M 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T14:55:34.287 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 173 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout:Install 11 Packages 2024-04-16T14:55:34.288 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:55:34.289 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 87 M 2024-04-16T14:55:34.289 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 813 M 2024-04-16T14:55:34.289 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:55:35.220 INFO:teuthology.orchestra.run.smithi097.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 662 kB/s | 217 kB 00:00 2024-04-16T14:55:35.379 INFO:teuthology.orchestra.run.smithi097.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-04-16T14:55:35.429 INFO:teuthology.orchestra.run.smithi097.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-16T14:55:35.454 INFO:teuthology.orchestra.run.smithi097.stdout:(4/11): libcephsqlite-19.0.0-3100.g3f7db60e.el9 308 kB/s | 173 kB 00:00 2024-04-16T14:55:35.471 INFO:teuthology.orchestra.run.smithi097.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-16T14:55:35.488 INFO:teuthology.orchestra.run.smithi097.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 90 kB 00:00 2024-04-16T14:55:35.705 INFO:teuthology.orchestra.run.smithi097.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 801 kB/s | 187 kB 00:00 2024-04-16T14:55:35.747 INFO:teuthology.orchestra.run.smithi097.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.8 MB/s | 469 kB 00:00 2024-04-16T14:55:35.939 INFO:teuthology.orchestra.run.smithi097.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 800 kB/s | 187 kB 00:00 2024-04-16T14:55:36.039 INFO:teuthology.orchestra.run.smithi097.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 417 kB/s | 122 kB 00:00 2024-04-16T14:55:39.598 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 31/34 2024-04-16T14:55:39.598 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-16T14:55:39.598 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-16T14:55:39.599 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-16T14:55:39.599 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-16T14:55:39.599 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-16T14:55:39.599 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-16T14:55:39.599 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-16T14:55:39.599 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:40.682 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/34 2024-04-16T14:55:40.707 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/34 2024-04-16T14:55:40.707 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:55:40.707 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T14:55:40.707 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T14:55:40.707 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T14:55:40.707 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:40.709 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T14:55:40.793 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T14:55:40.793 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T14:55:42.030 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T14:55:42.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/34 2024-04-16T14:55:42.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/34 2024-04-16T14:55:42.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 3/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 5/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_6 6/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 7/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3100.g3f7db60e.el9. 8/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3100.g3f7db60e.el9.x8 9/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 10/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 12/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 13/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T14:55:42.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 31/34 2024-04-16T14:55:42.032 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T14:55:42.033 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 33/34 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.203 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.204 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:43.205 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:55:43.444 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test 2024-04-16T14:55:44.031 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:03 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:55:44.198 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:55:44.198 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 84 M 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 173 k 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T14:55:44.199 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout:Install 11 Packages 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 87 M 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 813 M 2024-04-16T14:55:44.200 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:55:44.990 INFO:teuthology.orchestra.run.smithi186.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-16T14:55:45.057 INFO:teuthology.orchestra.run.smithi186.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-16T14:55:45.099 INFO:teuthology.orchestra.run.smithi186.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2024-04-16T14:55:45.141 INFO:teuthology.orchestra.run.smithi186.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-16T14:55:45.191 INFO:teuthology.orchestra.run.smithi186.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 90 kB 00:00 2024-04-16T14:55:45.392 INFO:teuthology.orchestra.run.smithi186.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-04-16T14:55:45.534 INFO:teuthology.orchestra.run.smithi186.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 3.2 MB/s | 469 kB 00:00 2024-04-16T14:55:45.575 INFO:teuthology.orchestra.run.smithi186.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-04-16T14:55:45.884 INFO:teuthology.orchestra.run.smithi186.stdout:(9/11): libcephsqlite-19.0.0-3100.g3f7db60e.el9 163 kB/s | 173 kB 00:01 2024-04-16T14:55:46.026 INFO:teuthology.orchestra.run.smithi186.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 270 kB/s | 122 kB 00:00 2024-04-16T14:56:49.478 INFO:teuthology.orchestra.run.smithi186.stdout:(11/11): ceph-test-19.0.0-3100.g3f7db60e.el9.x8 1.3 MB/s | 84 MB 01:04 2024-04-16T14:56:49.480 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:56:49.480 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.3 MB/s | 87 MB 01:05 2024-04-16T14:56:49.515 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:56:49.537 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:56:49.537 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:56:50.190 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:56:50.190 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:56:51.209 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:56:51.286 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T14:56:51.328 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T14:56:51.354 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T14:56:51.428 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T14:56:51.462 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T14:56:51.505 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T14:56:51.551 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T14:56:51.588 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T14:56:51.630 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T14:56:51.674 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T14:56:51.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 10/11 2024-04-16T14:57:03.015 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 10/11 2024-04-16T14:57:03.093 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T14:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T14:57:09.556 INFO:teuthology.orchestra.run.smithi186.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:09.557 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:57:09.885 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph 2024-04-16T14:57:10.487 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:29 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:57:10.642 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 6.5 k 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 2.2 M 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 1.6 M 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 249 k 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 5.0 M 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 71 M 2024-04-16T14:57:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T14:57:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T14:57:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout:Install 36 Packages 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:10.648 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 84 M 2024-04-16T14:57:10.649 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 922 M 2024-04-16T14:57:10.649 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:57:11.569 INFO:teuthology.orchestra.run.smithi186.stdout:(1/36): ceph-19.0.0-3100.g3f7db60e.el9.x86_64.r 16 kB/s | 6.5 kB 00:00 2024-04-16T14:57:12.661 INFO:teuthology.orchestra.run.smithi186.stdout:(2/36): ceph-mds-19.0.0-3100.g3f7db60e.el9.x86_ 1.5 MB/s | 2.2 MB 00:01 2024-04-16T14:57:14.245 INFO:teuthology.orchestra.run.smithi186.stdout:(3/36): ceph-mgr-19.0.0-3100.g3f7db60e.el9.x86_ 517 kB/s | 1.6 MB 00:03 2024-04-16T14:57:14.770 INFO:teuthology.orchestra.run.smithi186.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3100.g3f7d 474 kB/s | 249 kB 00:00 2024-04-16T14:57:14.887 INFO:teuthology.orchestra.run.smithi186.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-16T14:57:14.920 INFO:teuthology.orchestra.run.smithi186.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-04-16T14:57:14.954 INFO:teuthology.orchestra.run.smithi186.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-04-16T14:57:14.987 INFO:teuthology.orchestra.run.smithi186.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi186.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T14:57:15.130 INFO:teuthology.orchestra.run.smithi186.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-04-16T14:57:15.197 INFO:teuthology.orchestra.run.smithi186.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-04-16T14:57:15.239 INFO:teuthology.orchestra.run.smithi186.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-04-16T14:57:15.272 INFO:teuthology.orchestra.run.smithi186.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-04-16T14:57:15.306 INFO:teuthology.orchestra.run.smithi186.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-04-16T14:57:15.423 INFO:teuthology.orchestra.run.smithi186.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-04-16T14:57:15.515 INFO:teuthology.orchestra.run.smithi186.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-04-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi186.stdout:(17/36): ceph-mon-19.0.0-3100.g3f7db60e.el9.x86 1.2 MB/s | 5.0 MB 00:04 2024-04-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi186.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 290 kB/s | 43 kB 00:00 2024-04-16T14:57:16.125 INFO:teuthology.orchestra.run.smithi186.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-16T14:57:16.383 INFO:teuthology.orchestra.run.smithi186.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-04-16T14:57:16.442 INFO:teuthology.orchestra.run.smithi186.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-04-16T14:57:16.500 INFO:teuthology.orchestra.run.smithi186.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 304 kB/s | 18 kB 00:00 2024-04-16T14:57:16.559 INFO:teuthology.orchestra.run.smithi186.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-04-16T14:57:16.626 INFO:teuthology.orchestra.run.smithi186.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-04-16T14:57:16.701 INFO:teuthology.orchestra.run.smithi186.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 263 kB/s | 20 kB 00:00 2024-04-16T14:57:16.835 INFO:teuthology.orchestra.run.smithi186.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 347 kB/s | 46 kB 00:00 2024-04-16T14:57:16.968 INFO:teuthology.orchestra.run.smithi186.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 591 kB/s | 79 kB 00:00 2024-04-16T14:57:17.135 INFO:teuthology.orchestra.run.smithi186.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-16T14:57:17.177 INFO:teuthology.orchestra.run.smithi186.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-04-16T14:57:17.252 INFO:teuthology.orchestra.run.smithi186.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-16T14:57:17.303 INFO:teuthology.orchestra.run.smithi186.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-04-16T14:57:17.387 INFO:teuthology.orchestra.run.smithi186.stdout:(32/36): luarocks-3.9.2-1.el9.noarch.rpm 81 kB/s | 151 kB 00:01 2024-04-16T14:57:17.604 INFO:teuthology.orchestra.run.smithi186.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 765 kB/s | 230 kB 00:00 2024-04-16T14:57:17.670 INFO:teuthology.orchestra.run.smithi186.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 301 kB/s | 20 kB 00:00 2024-04-16T14:57:18.146 INFO:teuthology.orchestra.run.smithi186.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 563 kB/s | 427 kB 00:00 2024-04-16T14:57:34.119 INFO:teuthology.orchestra.run.smithi097.stdout:(11/11): ceph-test-19.0.0-3100.g3f7db60e.el9.x8 724 kB/s | 84 MB 01:59 2024-04-16T14:57:34.121 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:57:34.121 INFO:teuthology.orchestra.run.smithi097.stdout:Total 744 kB/s | 87 MB 01:59 2024-04-16T14:57:34.156 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:57:34.178 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:57:34.178 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:57:34.803 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:57:34.803 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:57:35.806 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:57:35.864 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T14:57:35.904 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T14:57:35.986 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T14:57:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T14:57:36.064 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T14:57:36.110 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T14:57:36.149 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T14:57:36.180 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T14:57:36.224 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T14:57:36.251 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 10/11 2024-04-16T14:57:39.952 INFO:teuthology.orchestra.run.smithi186.stdout:(36/36): ceph-osd-19.0.0-3100.g3f7db60e.el9.x86 2.6 MB/s | 71 MB 00:27 2024-04-16T14:57:39.957 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:57:39.957 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.9 MB/s | 84 MB 00:29 2024-04-16T14:57:39.984 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:57:40.029 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:57:40.030 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:57:40.771 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:57:40.772 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:57:42.370 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:57:42.429 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T14:57:42.462 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T14:57:42.490 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T14:57:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T14:57:42.605 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T14:57:42.638 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T14:57:42.730 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T14:57:42.808 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T14:57:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T14:57:42.938 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T14:57:42.992 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T14:57:43.030 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T14:57:43.079 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T14:57:43.134 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T14:57:43.332 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T14:57:43.376 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T14:57:43.413 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T14:57:43.449 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T14:57:43.479 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T14:57:43.556 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T14:57:43.590 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T14:57:43.623 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T14:57:43.664 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T14:57:43.699 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T14:57:43.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T14:57:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T14:57:43.872 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T14:57:43.946 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T14:57:44.083 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T14:57:44.210 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T14:57:44.405 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3100.g3f7db60e.el9. 31/36 2024-04-16T14:57:44.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/36 2024-04-16T14:57:44.463 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/36 2024-04-16T14:57:44.463 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:57:44.463 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T14:57:44.463 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T14:57:44.463 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T14:57:44.463 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:47.697 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 10/11 2024-04-16T14:57:47.775 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 11/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T14:57:53.802 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T14:57:53.803 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T14:57:53.803 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T14:57:53.803 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.6-16.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:57:54.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2024-04-16T14:57:55.091 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:17 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:57:55.248 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:57:55.250 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-04-16T14:57:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:57:55.250 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-04-16T14:57:55.250 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:57:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 6.5 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 2.2 M 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 1.6 M 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 249 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 5.0 M 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 71 M 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T14:57:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T14:57:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout:Install 36 Packages 2024-04-16T14:57:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:57:55.254 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 84 M 2024-04-16T14:57:55.254 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 922 M 2024-04-16T14:57:55.254 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:57:55.269 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 33/36 2024-04-16T14:57:55.297 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 33/36 2024-04-16T14:57:55.297 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:57:55.297 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T14:57:55.298 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T14:57:55.298 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T14:57:55.298 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:55.837 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 34/36 2024-04-16T14:57:55.861 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 34/36 2024-04-16T14:57:55.861 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:57:55.861 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T14:57:55.862 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T14:57:55.862 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T14:57:55.862 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:56.018 INFO:teuthology.orchestra.run.smithi097.stdout:(1/36): ceph-19.0.0-3100.g3f7db60e.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2024-04-16T14:57:56.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 35/36 2024-04-16T14:57:56.120 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 35/36 2024-04-16T14:57:56.120 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:57:56.120 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T14:57:56.120 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T14:57:56.120 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T14:57:56.120 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:57:56.185 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 36/36 2024-04-16T14:57:56.485 INFO:teuthology.orchestra.run.smithi097.stdout:(2/36): ceph-mgr-19.0.0-3100.g3f7db60e.el9.x86_ 2.1 MB/s | 1.6 MB 00:00 2024-04-16T14:57:58.610 INFO:teuthology.orchestra.run.smithi097.stdout:(3/36): ceph-mon-19.0.0-3100.g3f7db60e.el9.x86_ 1.9 MB/s | 5.0 MB 00:02 2024-04-16T14:57:58.785 INFO:teuthology.orchestra.run.smithi097.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3100.g3f7d 1.4 MB/s | 249 kB 00:00 2024-04-16T14:57:58.894 INFO:teuthology.orchestra.run.smithi097.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-04-16T14:57:58.927 INFO:teuthology.orchestra.run.smithi097.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.3 MB/s | 144 kB 00:00 2024-04-16T14:57:58.961 INFO:teuthology.orchestra.run.smithi097.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 944 kB/s | 32 kB 00:00 2024-04-16T14:57:58.995 INFO:teuthology.orchestra.run.smithi097.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T14:57:59.037 INFO:teuthology.orchestra.run.smithi097.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 925 kB/s | 39 kB 00:00 2024-04-16T14:57:59.087 INFO:teuthology.orchestra.run.smithi097.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-16T14:57:59.154 INFO:teuthology.orchestra.run.smithi097.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-04-16T14:57:59.213 INFO:teuthology.orchestra.run.smithi097.stdout:(12/36): ceph-mds-19.0.0-3100.g3f7db60e.el9.x86 643 kB/s | 2.2 MB 00:03 2024-04-16T14:57:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-04-16T14:57:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-16T14:57:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-04-16T14:57:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-04-16T14:57:59.422 INFO:teuthology.orchestra.run.smithi097.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 787 kB/s | 125 kB 00:00 2024-04-16T14:57:59.564 INFO:teuthology.orchestra.run.smithi097.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 724 kB/s | 151 kB 00:00 2024-04-16T14:57:59.590 INFO:teuthology.orchestra.run.smithi097.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 261 kB/s | 43 kB 00:00 2024-04-16T14:57:59.698 INFO:teuthology.orchestra.run.smithi097.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-16T14:57:59.740 INFO:teuthology.orchestra.run.smithi097.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-16T14:57:59.782 INFO:teuthology.orchestra.run.smithi097.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-04-16T14:57:59.824 INFO:teuthology.orchestra.run.smithi097.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-04-16T14:57:59.849 INFO:teuthology.orchestra.run.smithi097.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 345 kB/s | 23 kB 00:00 2024-04-16T14:57:59.875 INFO:teuthology.orchestra.run.smithi097.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-04-16T14:57:59.900 INFO:teuthology.orchestra.run.smithi097.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-04-16T14:57:59.942 INFO:teuthology.orchestra.run.smithi097.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 688 kB/s | 46 kB 00:00 2024-04-16T14:58:00.001 INFO:teuthology.orchestra.run.smithi097.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 785 kB/s | 79 kB 00:00 2024-04-16T14:58:00.043 INFO:teuthology.orchestra.run.smithi097.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-04-16T14:58:00.135 INFO:teuthology.orchestra.run.smithi097.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 980 kB/s | 90 kB 00:00 2024-04-16T14:58:00.185 INFO:teuthology.orchestra.run.smithi097.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-04-16T14:58:00.210 INFO:teuthology.orchestra.run.smithi097.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-04-16T14:58:00.344 INFO:teuthology.orchestra.run.smithi097.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-16T14:58:00.386 INFO:teuthology.orchestra.run.smithi097.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-04-16T14:58:00.461 INFO:teuthology.orchestra.run.smithi097.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-04-16T14:58:03.045 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 36/36 2024-04-16T14:58:03.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 3/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 5/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3100.g3f7db60e.el9. 6/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T14:58:03.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T14:58:03.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T14:58:03.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T14:58:03.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T14:58:03.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T14:58:03.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T14:58:03.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T14:58:03.837 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T14:58:03.838 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T14:58:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:03.840 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:04.107 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base 2024-04-16T14:58:04.228 INFO:teuthology.orchestra.run.smithi097.stdout:(36/36): ceph-osd-19.0.0-3100.g3f7db60e.el9.x86 9.2 MB/s | 71 MB 00:07 2024-04-16T14:58:04.234 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:04.234 INFO:teuthology.orchestra.run.smithi097.stdout:Total 9.4 MB/s | 84 MB 00:08 2024-04-16T14:58:04.260 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:58:04.693 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:23 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:04.794 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:04.848 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:04.849 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:04.849 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:04.914 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install cephadm 2024-04-16T14:58:05.023 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:58:05.024 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:58:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:24 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:05.663 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:05.664 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:05.664 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:05.664 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:05.664 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 762 k 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 762 k 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 769 k 2024-04-16T14:58:05.665 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:06.631 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:58:06.697 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T14:58:06.729 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T14:58:06.766 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T14:58:06.824 INFO:teuthology.orchestra.run.smithi186.stdout:cephadm-19.0.0-3100.g3f7db60e.el9.noarch.rpm 658 kB/s | 762 kB 00:01 2024-04-16T14:58:06.824 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:06.824 INFO:teuthology.orchestra.run.smithi186.stdout:Total 657 kB/s | 762 kB 00:01 2024-04-16T14:58:06.825 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:06.831 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:06.831 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:06.836 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T14:58:06.837 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:06.837 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T14:58:06.914 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T14:58:06.914 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T14:58:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T14:58:07.134 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T14:58:07.204 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T14:58:07.259 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T14:58:07.298 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T14:58:07.355 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T14:58:07.398 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:07.411 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T14:58:07.496 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:07.596 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T14:58:07.644 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T14:58:07.682 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T14:58:07.717 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T14:58:07.748 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T14:58:07.823 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T14:58:07.858 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T14:58:07.882 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:07.891 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T14:58:07.931 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T14:58:07.966 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T14:58:08.001 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T14:58:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T14:58:08.140 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T14:58:08.156 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:08.156 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:08.156 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:08.156 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:08.156 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:08.156 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:08.213 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T14:58:08.338 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T14:58:08.352 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T14:58:08.475 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T14:58:08.675 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3100.g3f7db60e.el9. 31/36 2024-04-16T14:58:08.704 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/36 2024-04-16T14:58:08.729 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 32/36 2024-04-16T14:58:08.729 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:08.729 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T14:58:08.729 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T14:58:08.729 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T14:58:08.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:27 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 157 k 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 157 k 2024-04-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 640 k 2024-04-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:09.605 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-19.0.0-3100.g3f7db6 309 kB/s | 157 kB 00:00 2024-04-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi186.stdout:Total 308 kB/s | 157 kB 00:00 2024-04-16T14:58:09.606 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:09.616 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:09.616 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:09.669 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:09.670 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:09.812 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:09.873 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e. 1/1 2024-04-16T14:58:09.899 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e. 1/1 2024-04-16T14:58:09.899 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:09.899 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T14:58:09.899 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:10.866 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e. 1/1 2024-04-16T14:58:10.866 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:10.866 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:10.866 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:10.867 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:10.867 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:11.110 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr 2024-04-16T14:58:11.705 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:30 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:11.804 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:11.856 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:11.857 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:11.858 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:11.921 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T14:58:12.513 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:31 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:12.670 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 3.5 M 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 34 k 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 16 k 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T14:58:12.671 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout:Install 11 Packages 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:12.672 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 7.6 M 2024-04-16T14:58:12.673 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 100 M 2024-04-16T14:58:12.673 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:13.518 INFO:teuthology.orchestra.run.smithi186.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3100.g3f 102 kB/s | 34 kB 00:00 2024-04-16T14:58:13.760 INFO:teuthology.orchestra.run.smithi186.stdout:(2/11): python3-protobuf-3.14.0-13.el9.noarch.r 985 kB/s | 238 kB 00:00 2024-04-16T14:58:13.785 INFO:teuthology.orchestra.run.smithi186.stdout:(3/11): ceph-prometheus-alerts-19.0.0-3100.g3f7 27 kB/s | 16 kB 00:00 2024-04-16T14:58:13.994 INFO:teuthology.orchestra.run.smithi186.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-04-16T14:58:14.302 INFO:teuthology.orchestra.run.smithi186.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-04-16T14:58:14.428 INFO:teuthology.orchestra.run.smithi186.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 853 kB/s | 548 kB 00:00 2024-04-16T14:58:14.544 INFO:teuthology.orchestra.run.smithi186.stdout:(7/11): ceph-mgr-dashboard-19.0.0-3100.g3f7db60 2.6 MB/s | 3.5 MB 00:01 2024-04-16T14:58:14.570 INFO:teuthology.orchestra.run.smithi186.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-04-16T14:58:14.678 INFO:teuthology.orchestra.run.smithi186.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-04-16T14:58:14.870 INFO:teuthology.orchestra.run.smithi186.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.6 MB/s | 2.0 MB 00:00 2024-04-16T14:58:14.895 INFO:teuthology.orchestra.run.smithi186.stdout:(11/11): python3-repoze-lru-0.7-16.el9.noarch.r 88 kB/s | 31 kB 00:00 2024-04-16T14:58:14.900 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:14.901 INFO:teuthology.orchestra.run.smithi186.stdout:Total 3.4 MB/s | 7.6 MB 00:02 2024-04-16T14:58:14.919 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:14.936 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:14.936 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:15.105 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:15.105 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:15.599 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:15.727 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T14:58:15.766 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T14:58:15.875 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T14:58:16.010 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T14:58:16.069 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T14:58:16.154 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T14:58:16.196 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T14:58:16.258 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T14:58:16.290 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3100.g3f7db60e.el9 9/11 2024-04-16T14:58:17.342 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3100.g3f7db60e.el 10/11 2024-04-16T14:58:17.382 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noa 11/11 2024-04-16T14:58:18.661 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noa 11/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3100.g3f7db60e.el 1/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noa 2/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3100.g3f7db60e.el9 3/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T14:58:18.662 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T14:58:19.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T14:58:19.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T14:58:19.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T14:58:19.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T14:58:19.002 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:19.002 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:19.266 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T14:58:19.529 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 33/36 2024-04-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 33/36 2024-04-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:19.887 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:38 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:20.052 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:20.053 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-04-16T14:58:20.053 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:20.053 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-04-16T14:58:20.053 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:20.053 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 7.4 M 2024-04-16T14:58:20.053 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:20.054 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:20.055 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-04-16T14:58:20.055 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-04-16T14:58:20.055 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:20.055 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 37 M 2024-04-16T14:58:20.055 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 209 M 2024-04-16T14:58:20.055 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:20.097 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 34/36 2024-04-16T14:58:20.123 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 34/36 2024-04-16T14:58:20.123 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:20.123 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T14:58:20.123 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T14:58:20.123 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T14:58:20.123 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:20.373 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 35/36 2024-04-16T14:58:20.398 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 35/36 2024-04-16T14:58:20.398 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:20.398 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T14:58:20.399 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T14:58:20.399 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T14:58:20.399 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:20.467 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 36/36 2024-04-16T14:58:20.629 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 134 kB/s | 32 kB 00:00 2024-04-16T14:58:20.671 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-04-16T14:58:20.755 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-04-16T14:58:20.780 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 319 kB/s | 34 kB 00:00 2024-04-16T14:58:21.689 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-04-16T14:58:21.756 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.9 MB/s | 4.8 MB 00:00 2024-04-16T14:58:21.781 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-04-16T14:58:22.298 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-16T14:58:22.623 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-04-16T14:58:22.648 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 559 kB/s | 195 kB 00:00 2024-04-16T14:58:22.932 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 796 kB/s | 245 kB 00:00 2024-04-16T14:58:23.040 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 2.8 MB/s | 7.4 MB 00:02 2024-04-16T14:58:23.042 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:23.042 INFO:teuthology.orchestra.run.smithi186.stdout:Total 12 MB/s | 37 MB 00:02 2024-04-16T14:58:23.084 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:23.109 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:23.109 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:23.468 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:23.468 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:24.173 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:24.238 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T14:58:24.270 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T14:58:24.364 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T14:58:24.398 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T14:58:24.575 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T14:58:24.606 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T14:58:24.682 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T14:58:25.345 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T14:58:25.459 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T14:58:26.695 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 36/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 3/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 4/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 5/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3100.g3f7db60e.el9. 6/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T14:58:27.265 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T14:58:27.266 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T14:58:27.267 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T14:58:27.706 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T14:58:27.739 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db 12/12 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T14:58:28.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T14:58:28.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T14:58:28.110 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T14:58:28.110 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T14:58:28.110 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T14:58:28.110 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:28.110 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:28.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2024-04-16T14:58:28.939 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:50 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:29.039 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:29.091 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:29.092 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:58:29.092 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:29.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2024-04-16T14:58:29.745 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:51 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:29.896 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:29.897 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:58:29.897 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:58:29.897 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 762 k 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 762 k 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 769 k 2024-04-16T14:58:29.898 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db 12/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db 1/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T14:58:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T14:58:30.443 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-19.0.0-3100.g3f7db60e.el9.noarch.rpm 1.4 MB/s | 762 kB 00:00 2024-04-16T14:58:30.444 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:30.444 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.4 MB/s | 762 kB 00:00 2024-04-16T14:58:30.444 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:58:30.449 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:58:30.449 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:58:30.455 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:58:30.455 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:58:30.533 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:58:30.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T14:58:30.820 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:30.820 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:31.011 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:31.116 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:31.196 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-rook 2024-04-16T14:58:31.487 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:31.760 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/1 2024-04-16T14:58:31.760 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:31.760 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:58:31.760 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:31.760 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:31.761 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:31.802 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:50 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:31.959 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:31.960 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-16T14:58:31.960 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:31.960 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-16T14:58:31.960 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:31.960 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 49 k 2024-04-16T14:58:31.960 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:31.961 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:31.962 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-16T14:58:31.962 INFO:teuthology.orchestra.run.smithi186.stdout:Install 13 Packages 2024-04-16T14:58:31.962 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:31.962 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.0 M 2024-04-16T14:58:31.962 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-04-16T14:58:31.963 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:31.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T14:58:32.282 INFO:teuthology.orchestra.run.smithi186.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-04-16T14:58:32.316 INFO:teuthology.orchestra.run.smithi186.stdout:(2/13): ceph-mgr-rook-19.0.0-3100.g3f7db60e.el9 262 kB/s | 49 kB 00:00 2024-04-16T14:58:32.341 INFO:teuthology.orchestra.run.smithi186.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 114 kB/s | 24 kB 00:00 2024-04-16T14:58:32.425 INFO:teuthology.orchestra.run.smithi186.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-04-16T14:58:32.442 INFO:teuthology.orchestra.run.smithi186.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-04-16T14:58:32.467 INFO:teuthology.orchestra.run.smithi186.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 211 kB 00:00 2024-04-16T14:58:32.492 INFO:teuthology.orchestra.run.smithi186.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 641 kB/s | 43 kB 00:00 2024-04-16T14:58:32.585 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:54 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:32.744 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:32.744 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 157 k 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 157 k 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 640 k 2024-04-16T14:58:32.745 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:58:32.784 INFO:teuthology.orchestra.run.smithi186.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-04-16T14:58:32.834 INFO:teuthology.orchestra.run.smithi186.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 82 kB/s | 32 kB 00:00 2024-04-16T14:58:33.043 INFO:teuthology.orchestra.run.smithi186.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 368 kB/s | 203 kB 00:00 2024-04-16T14:58:33.068 INFO:teuthology.orchestra.run.smithi186.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-04-16T14:58:33.208 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-19.0.0-3100.g3f7db6 339 kB/s | 157 kB 00:00 2024-04-16T14:58:33.209 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:33.209 INFO:teuthology.orchestra.run.smithi097.stdout:Total 338 kB/s | 157 kB 00:00 2024-04-16T14:58:33.210 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:58:33.219 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:58:33.219 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:58:33.252 INFO:teuthology.orchestra.run.smithi186.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-04-16T14:58:33.273 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:58:33.273 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:58:33.410 INFO:teuthology.orchestra.run.smithi186.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-04-16T14:58:33.413 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:33.413 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-04-16T14:58:33.413 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:33.424 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:58:33.428 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:33.428 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:33.484 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e. 1/1 2024-04-16T14:58:33.510 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e. 1/1 2024-04-16T14:58:33.510 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:33.510 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T14:58:33.510 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:33.527 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:33.527 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:34.039 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:34.196 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T14:58:34.252 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T14:58:34.307 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T14:58:34.337 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T14:58:34.378 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T14:58:34.471 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T14:58:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e. 1/1 2024-04-16T14:58:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:34.476 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:58:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:34.476 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:34.580 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T14:58:34.630 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T14:58:34.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2024-04-16T14:58:35.344 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T14:58:35.356 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:57 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:35.395 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T14:58:35.420 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T14:58:35.456 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:35.468 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T14:58:35.490 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 13/13 2024-04-16T14:58:35.509 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:35.510 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:58:35.510 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:35.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T14:58:36.175 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:58 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 13/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 1/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T14:58:36.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T14:58:36.252 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T14:58:36.252 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T14:58:36.252 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T14:58:36.329 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:36.330 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-04-16T14:58:36.330 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:58:36.330 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-04-16T14:58:36.330 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:58:36.330 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 3.5 M 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 34 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 16 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:58:36.331 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-04-16T14:58:36.332 INFO:teuthology.orchestra.run.smithi097.stdout:Install 11 Packages 2024-04-16T14:58:36.332 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:36.332 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 7.6 M 2024-04-16T14:58:36.332 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 100 M 2024-04-16T14:58:36.332 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:58:36.627 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T14:58:36.627 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:36.627 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:36.627 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:36.627 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:36.628 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:36.868 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T14:58:36.938 INFO:teuthology.orchestra.run.smithi097.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3100.g3f7 68 kB/s | 16 kB 00:00 2024-04-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi097.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3100.g3f 132 kB/s | 34 kB 00:00 2024-04-16T14:58:37.080 INFO:teuthology.orchestra.run.smithi097.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-04-16T14:58:37.155 INFO:teuthology.orchestra.run.smithi097.stdout:(4/11): ceph-mgr-dashboard-19.0.0-3100.g3f7db60 7.7 MB/s | 3.5 MB 00:00 2024-04-16T14:58:37.247 INFO:teuthology.orchestra.run.smithi097.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-04-16T14:58:37.457 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:56 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:37.464 INFO:teuthology.orchestra.run.smithi097.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 68 kB/s | 19 kB 00:00 2024-04-16T14:58:37.598 INFO:teuthology.orchestra.run.smithi097.stdout:(7/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-04-16T14:58:37.611 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 142 k 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T14:58:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.8 M 2024-04-16T14:58:37.613 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 33 M 2024-04-16T14:58:37.614 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:37.665 INFO:teuthology.orchestra.run.smithi097.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 463 kB/s | 31 kB 00:00 2024-04-16T14:58:37.690 INFO:teuthology.orchestra.run.smithi097.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 640 kB/s | 144 kB 00:00 2024-04-16T14:58:37.757 INFO:teuthology.orchestra.run.smithi097.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-16T14:58:37.916 INFO:teuthology.orchestra.run.smithi097.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.2 MB/s | 2.0 MB 00:00 2024-04-16T14:58:37.920 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:37.920 INFO:teuthology.orchestra.run.smithi097.stdout:Total 4.8 MB/s | 7.6 MB 00:01 2024-04-16T14:58:37.939 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:58:37.956 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:58:37.956 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:58:38.124 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:58:38.125 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:58:38.213 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 896 kB/s | 248 kB 00:00 2024-04-16T14:58:38.363 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-16T14:58:38.388 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3100.g3f7db60e.e 314 kB/s | 142 kB 00:00 2024-04-16T14:58:38.489 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 462 kB/s | 58 kB 00:00 2024-04-16T14:58:38.514 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-04-16T14:58:38.539 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-04-16T14:58:38.542 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:38.542 INFO:teuthology.orchestra.run.smithi186.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-04-16T14:58:38.555 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:38.565 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:38.565 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:38.607 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:58:38.656 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:38.656 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:38.697 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T14:58:38.736 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T14:58:38.834 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T14:58:38.950 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T14:58:39.022 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T14:58:39.108 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T14:58:39.113 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T14:58:39.149 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T14:58:39.157 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T14:58:39.178 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T14:58:39.210 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3100.g3f7db60e.el9 9/11 2024-04-16T14:58:39.680 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T14:58:39.761 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T14:58:39.823 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T14:58:39.853 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 6/6 2024-04-16T14:58:40.256 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3100.g3f7db60e.el 10/11 2024-04-16T14:58:40.294 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noa 11/11 2024-04-16T14:58:40.689 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 6/6 2024-04-16T14:58:40.689 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/6 2024-04-16T14:58:40.689 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T14:58:40.689 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T14:58:40.689 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T14:58:40.689 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T14:58:40.994 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T14:58:40.995 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T14:58:40.995 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:40.995 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:41.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse 2024-04-16T14:58:41.627 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noa 11/11 2024-04-16T14:58:41.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3100.g3f7db60e.el 1/11 2024-04-16T14:58:41.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noa 2/11 2024-04-16T14:58:41.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3100.g3f7db60e.el9 3/11 2024-04-16T14:58:41.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T14:58:41.628 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T14:58:41.628 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T14:58:41.628 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T14:58:41.628 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T14:58:41.628 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T14:58:41.628 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T14:58:41.838 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:00 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T14:58:41.953 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:41.954 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:41.994 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 871 k 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 949 k 2024-04-16T14:58:41.995 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 4.0 M 2024-04-16T14:58:41.996 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:42.143 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 533 kB/s | 79 kB 00:00 2024-04-16T14:58:42.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T14:58:42.377 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-fuse-19.0.0-3100.g3f7db60e.el9.x86_ 2.2 MB/s | 871 kB 00:00 2024-04-16T14:58:42.378 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:42.378 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.4 MB/s | 949 kB 00:00 2024-04-16T14:58:42.379 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:42.401 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:42.401 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:42.474 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:42.475 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:42.665 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:42.804 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:04 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:42.820 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T14:58:42.894 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/2 2024-04-16T14:58:42.959 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:42.960 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-04-16T14:58:42.960 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:58:42.960 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-04-16T14:58:42.960 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 7.4 M 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T14:58:42.961 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:42.962 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:58:42.962 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-04-16T14:58:42.962 INFO:teuthology.orchestra.run.smithi097.stdout:Install 12 Packages 2024-04-16T14:58:42.962 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:42.962 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 37 M 2024-04-16T14:58:42.962 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 209 M 2024-04-16T14:58:42.963 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:58:43.507 INFO:teuthology.orchestra.run.smithi097.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-04-16T14:58:43.540 INFO:teuthology.orchestra.run.smithi097.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2024-04-16T14:58:43.574 INFO:teuthology.orchestra.run.smithi097.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-16T14:58:43.580 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/2 2024-04-16T14:58:43.580 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/2 2024-04-16T14:58:43.641 INFO:teuthology.orchestra.run.smithi097.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-04-16T14:58:43.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T14:58:43.877 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:43.877 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:43.877 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T14:58:43.877 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:43.877 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:43.983 INFO:teuthology.orchestra.run.smithi097.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-04-16T14:58:44.050 INFO:teuthology.orchestra.run.smithi097.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-04-16T14:58:44.075 INFO:teuthology.orchestra.run.smithi097.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-04-16T14:58:44.138 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-volume 2024-04-16T14:58:44.509 INFO:teuthology.orchestra.run.smithi097.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-04-16T14:58:44.551 INFO:teuthology.orchestra.run.smithi097.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-16T14:58:44.702 INFO:teuthology.orchestra.run.smithi097.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-04-16T14:58:44.727 INFO:teuthology.orchestra.run.smithi097.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 900 kB/s | 195 kB 00:00 2024-04-16T14:58:44.756 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:03 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:44.916 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:44.916 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 273 k 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-04-16T14:58:44.917 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:44.918 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 342 k 2024-04-16T14:58:44.918 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.5 M 2024-04-16T14:58:44.918 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:45.040 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 564 kB/s | 69 kB 00:00 2024-04-16T14:58:45.269 INFO:teuthology.orchestra.run.smithi097.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 3.9 MB/s | 7.4 MB 00:01 2024-04-16T14:58:45.271 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:45.271 INFO:teuthology.orchestra.run.smithi097.stdout:Total 16 MB/s | 37 MB 00:02 2024-04-16T14:58:45.314 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:58:45.315 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-volume-19.0.0-3100.g3f7db60e.el9.no 687 kB/s | 273 kB 00:00 2024-04-16T14:58:45.317 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:45.317 INFO:teuthology.orchestra.run.smithi186.stdout:Total 856 kB/s | 342 kB 00:00 2024-04-16T14:58:45.317 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:45.325 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:45.325 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:45.338 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:58:45.338 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:58:45.363 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:45.363 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:45.567 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:45.687 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:58:45.687 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:58:45.755 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T14:58:45.785 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 2/2 2024-04-16T14:58:45.802 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 2/2 2024-04-16T14:58:45.802 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:58:45.802 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T14:58:45.802 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:46.421 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:58:46.500 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T14:58:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T14:58:46.586 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 1/2 2024-04-16T14:58:46.625 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T14:58:46.659 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T14:58:46.846 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T14:58:46.884 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:46.925 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T14:58:47.188 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel 2024-04-16T14:58:47.618 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T14:58:47.721 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T14:58:47.791 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:06 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:47.944 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 125 k 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:58:47.945 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:47.946 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 125 k 2024-04-16T14:58:47.946 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 445 k 2024-04-16T14:58:47.946 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:48.956 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T14:58:49.440 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-19.0.0-3100.g3f7db60e.el9.x86_64 84 kB/s | 125 kB 00:01 2024-04-16T14:58:49.440 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:49.441 INFO:teuthology.orchestra.run.smithi186.stdout:Total 83 kB/s | 125 kB 00:01 2024-04-16T14:58:49.441 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:49.446 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:49.446 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:49.502 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:49.502 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:49.780 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:49.986 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T14:58:50.018 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db 12/12 2024-04-16T14:58:50.254 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:50.532 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:50.532 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:50.532 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:50.532 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:50.532 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:50.532 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:50.759 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2 2024-04-16T14:58:51.377 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:10 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:51.482 INFO:teuthology.orchestra.run.smithi186.stdout:Package libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:51.535 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:51.536 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:51.536 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:51.599 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs-devel 2024-04-16T14:58:52.199 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:11 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:52.353 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 32 k 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 32 k 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 142 k 2024-04-16T14:58:52.354 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:52.602 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs-devel-19.0.0-3100.g3f7db60e.el9.x86_6 128 kB/s | 32 kB 00:00 2024-04-16T14:58:52.603 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:52.603 INFO:teuthology.orchestra.run.smithi186.stdout:Total 127 kB/s | 32 kB 00:00 2024-04-16T14:58:52.603 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:52.606 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:52.606 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db 12/12 2024-04-16T14:58:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db 1/12 2024-04-16T14:58:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T14:58:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T14:58:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:52.723 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:52.851 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:53.081 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T14:58:53.081 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T14:58:53.082 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:53.083 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:53.244 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:53.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-rook 2024-04-16T14:58:53.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:53.499 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:53.499 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:58:53.499 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:58:53.499 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:53.499 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:53.675 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2 2024-04-16T14:58:54.091 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:16 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:58:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:58:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-04-16T14:58:54.248 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:58:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-04-16T14:58:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:58:54.248 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 49 k 2024-04-16T14:58:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-04-16T14:58:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2024-04-16T14:58:54.250 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:54.250 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 2.0 M 2024-04-16T14:58:54.250 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2024-04-16T14:58:54.250 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:58:54.271 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:13 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:54.372 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:54.424 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:54.425 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:54.425 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:54.490 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1 2024-04-16T14:58:54.527 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-04-16T14:58:54.552 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 95 kB/s | 18 kB 00:00 2024-04-16T14:58:54.653 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2024-04-16T14:58:54.678 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-04-16T14:58:54.712 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-04-16T14:58:54.737 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 728 kB/s | 43 kB 00:00 2024-04-16T14:58:54.762 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): ceph-mgr-rook-19.0.0-3100.g3f7db60e.el9 123 kB/s | 49 kB 00:00 2024-04-16T14:58:54.929 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-04-16T14:58:54.975 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 63 kB/s | 15 kB 00:00 2024-04-16T14:58:55.079 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 569 kB/s | 59 kB 00:00 2024-04-16T14:58:55.084 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:14 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:55.138 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 613 kB/s | 203 kB 00:00 2024-04-16T14:58:55.186 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:55.197 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-04-16T14:58:55.239 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:55.240 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:55.240 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:55.294 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados 2024-04-16T14:58:55.895 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:14 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:55.913 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.0 MB/s | 1.0 MB 00:00 2024-04-16T14:58:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-04-16T14:58:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:58:55.932 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:58:55.932 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:58:55.997 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:56.024 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:58:56.024 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:58:56.049 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:56.050 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:56.051 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:56.112 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw 2024-04-16T14:58:56.541 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:58:56.705 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:15 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T14:58:56.773 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T14:58:56.807 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:56.831 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T14:58:56.859 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:56.868 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T14:58:56.909 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T14:58:56.919 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs 2024-04-16T14:58:57.003 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T14:58:57.103 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T14:58:57.160 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T14:58:57.513 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:16 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:57.616 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:57.669 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:57.670 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:57.670 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:57.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd 2024-04-16T14:58:57.879 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T14:58:57.925 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T14:58:57.959 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T14:58:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T14:58:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 13/13 2024-04-16T14:58:58.329 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:17 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:58.432 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:58:58.486 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:58.487 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-16T14:58:58.487 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:58:58.544 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse 2024-04-16T14:58:58.787 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 13/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 1/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T14:58:58.788 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T14:58:59.139 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:18 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:58:59.143 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T14:58:59.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:59.143 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:58:59.143 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:58:59.143 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T14:58:59.143 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:58:59.144 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:58:59.294 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 90 k 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:59.295 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:58:59.296 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:58:59.296 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:58:59.296 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:58:59.296 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 90 k 2024-04-16T14:58:59.296 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 313 k 2024-04-16T14:58:59.296 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:58:59.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T14:58:59.497 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-19.0.0-3100.g3f7db60e.el9.x86_64.rpm 447 kB/s | 90 kB 00:00 2024-04-16T14:58:59.498 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:58:59.498 INFO:teuthology.orchestra.run.smithi186.stdout:Total 443 kB/s | 90 kB 00:00 2024-04-16T14:58:59.498 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:58:59.507 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:58:59.507 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:58:59.563 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:58:59.563 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:58:59.701 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:58:59.797 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:58:59.996 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:21 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:00.153 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:00.153 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 142 k 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T14:59:00.154 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 6.8 M 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 33 M 2024-04-16T14:59:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:00.167 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:00.473 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:00.474 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:00.474 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:59:00.474 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:00.474 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:00.474 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:59:00.709 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror 2024-04-16T14:59:00.902 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-04-16T14:59:01.019 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3100.g3f7db60e.e 543 kB/s | 142 kB 00:00 2024-04-16T14:59:01.322 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:20 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:59:01.452 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.3 MB/s | 5.8 MB 00:00 2024-04-16T14:59:01.478 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 126 kB/s | 58 kB 00:00 2024-04-16T14:59:01.483 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 3.4 M 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:59:01.484 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:01.485 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 3.4 M 2024-04-16T14:59:01.485 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 29 M 2024-04-16T14:59:01.485 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:59:01.778 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 626 kB/s | 548 kB 00:00 2024-04-16T14:59:01.853 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 127 kB/s | 51 kB 00:00 2024-04-16T14:59:01.856 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:01.856 INFO:teuthology.orchestra.run.smithi097.stdout:Total 4.0 MB/s | 6.8 MB 00:01 2024-04-16T14:59:01.869 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:01.879 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:01.880 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:01.970 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:01.970 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:02.275 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:02.450 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T14:59:02.496 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T14:59:02.660 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-19.0.0-3100.g3f7db60e.el9.x86_64.rpm 2.9 MB/s | 3.4 MB 00:01 2024-04-16T14:59:02.660 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:02.661 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.9 MB/s | 3.4 MB 00:01 2024-04-16T14:59:02.661 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:59:02.672 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:59:02.672 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:59:02.747 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:59:02.748 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:59:03.016 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T14:59:03.098 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T14:59:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T14:59:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 6/6 2024-04-16T14:59:03.302 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:59:03.360 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:03.386 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:03.386 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:59:03.386 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T14:59:03.387 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T14:59:03.387 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T14:59:03.387 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 6/6 2024-04-16T14:59:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 1/6 2024-04-16T14:59:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T14:59:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T14:59:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T14:59:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T14:59:04.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T14:59:04.289 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:04.289 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:04.290 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:04.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2024-04-16T14:59:04.713 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:04.713 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:04.713 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:59:04.713 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:04.713 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:04.713 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:59:04.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd 2024-04-16T14:59:05.152 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:27 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:05.310 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 871 k 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 949 k 2024-04-16T14:59:05.311 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 4.0 M 2024-04-16T14:59:05.312 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:05.497 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 425 kB/s | 79 kB 00:00 2024-04-16T14:59:05.576 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:24 ago on Tue 16 Apr 2024 02:54:41 PM UTC. 2024-04-16T14:59:05.730 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 175 k 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-16T14:59:05.731 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:05.732 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 175 k 2024-04-16T14:59:05.732 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 673 k 2024-04-16T14:59:05.732 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-16T14:59:05.789 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-fuse-19.0.0-3100.g3f7db60e.el9.x86_ 1.8 MB/s | 871 kB 00:00 2024-04-16T14:59:05.791 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:05.791 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.9 MB/s | 949 kB 00:00 2024-04-16T14:59:05.791 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:05.814 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:05.814 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:05.888 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:05.889 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:05.989 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-19.0.0-3100.g3f7db60e.el9.x86_64.rpm 683 kB/s | 175 kB 00:00 2024-04-16T14:59:05.989 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:05.990 INFO:teuthology.orchestra.run.smithi186.stdout:Total 679 kB/s | 175 kB 00:00 2024-04-16T14:59:05.990 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-16T14:59:06.000 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-16T14:59:06.000 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-16T14:59:06.056 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-16T14:59:06.056 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-16T14:59:06.070 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:06.216 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-16T14:59:06.224 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T14:59:06.289 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/2 2024-04-16T14:59:06.322 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:06.765 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2/2 2024-04-16T14:59:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/2 2024-04-16T14:59:07.057 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:07.057 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:07.057 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-16T14:59:07.057 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:07.057 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:07.057 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-16T14:59:07.280 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T14:59:07.281 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:07.281 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:07.281 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T14:59:07.281 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:07.281 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:07.319 DEBUG:teuthology.parallel:result is None 2024-04-16T14:59:07.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-volume 2024-04-16T14:59:08.163 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:30 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:08.322 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-3100.g3f7db60e.el9 ceph-noarch 273 k 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 342 k 2024-04-16T14:59:08.323 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.5 M 2024-04-16T14:59:08.324 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): ceph-volume-19.0.0-3100.g3f7db60e.el9.no 1.2 MB/s | 273 kB 00:00 2024-04-16T14:59:08.594 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 254 kB/s | 69 kB 00:00 2024-04-16T14:59:08.596 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:08.596 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.2 MB/s | 342 kB 00:00 2024-04-16T14:59:08.596 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:08.604 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:08.604 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:08.883 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:09.057 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T14:59:09.097 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 2/2 2024-04-16T14:59:09.114 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 2/2 2024-04-16T14:59:09.114 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:59:09.114 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T14:59:09.114 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:09.855 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 1/2 2024-04-16T14:59:10.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T14:59:10.182 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:10.182 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:10.183 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-3100.g3f7db60e.el9.noarch 2024-04-16T14:59:10.183 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T14:59:10.183 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:10.183 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:10.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2024-04-16T14:59:11.100 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:33 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:11.253 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:11.254 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:11.254 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-04-16T14:59:11.254 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:11.254 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 125 k 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 125 k 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 445 k 2024-04-16T14:59:11.255 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:12.152 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-19.0.0-3100.g3f7db60e.el9.x86_64 139 kB/s | 125 kB 00:00 2024-04-16T14:59:12.152 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:12.153 INFO:teuthology.orchestra.run.smithi097.stdout:Total 139 kB/s | 125 kB 00:00 2024-04-16T14:59:12.153 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:12.159 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:12.159 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:12.217 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:12.218 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:12.359 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:12.459 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:12.946 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:13.198 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:13.198 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:13.198 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:13.198 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:13.198 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:13.198 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:13.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2024-04-16T14:59:14.077 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:36 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:14.181 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:14.233 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:14.234 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:14.234 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:14.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2024-04-16T14:59:14.888 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:36 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:15.039 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:15.039 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 32 k 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 32 k 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 142 k 2024-04-16T14:59:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:15.214 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-19.0.0-3100.g3f7db60e.el9.x86_6 182 kB/s | 32 kB 00:00 2024-04-16T14:59:15.214 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:15.215 INFO:teuthology.orchestra.run.smithi097.stdout:Total 181 kB/s | 32 kB 00:00 2024-04-16T14:59:15.215 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:15.218 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:15.218 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:15.242 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:15.242 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:15.333 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:15.437 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:15.820 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:16.080 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:16.080 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:16.080 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:16.080 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:16.081 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:16.081 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:16.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2024-04-16T14:59:16.853 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:38 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:16.957 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:17.011 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:17.012 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:17.012 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:17.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2024-04-16T14:59:17.675 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:39 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:17.778 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:17.832 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:17.833 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:17.833 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:17.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2024-04-16T14:59:18.496 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:40 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:18.598 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:18.650 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:18.651 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:18.651 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:18.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2024-04-16T14:59:19.320 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:41 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:19.421 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:19.473 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:19.474 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:19.474 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:19.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2024-04-16T14:59:20.132 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:42 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:20.232 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:20.284 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:20.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2024-04-16T14:59:20.960 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:42 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:21.069 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 is already installed. 2024-04-16T14:59:21.124 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:21.125 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-04-16T14:59:21.125 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:21.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2024-04-16T14:59:21.821 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:43 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:21.982 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:21.983 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:21.983 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-04-16T14:59:21.983 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:21.983 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:21.983 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 90 k 2024-04-16T14:59:21.983 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 90 k 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 313 k 2024-04-16T14:59:21.984 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:23.376 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-19.0.0-3100.g3f7db60e.el9.x86_64.rpm 65 kB/s | 90 kB 00:01 2024-04-16T14:59:23.377 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:23.377 INFO:teuthology.orchestra.run.smithi097.stdout:Total 64 kB/s | 90 kB 00:01 2024-04-16T14:59:23.377 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:23.388 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:23.388 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:23.447 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:23.447 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:23.590 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:23.699 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:24.399 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:24.400 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:24.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2024-04-16T14:59:25.270 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:47 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:25.429 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 3.4 M 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:25.430 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:25.431 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:59:25.431 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:25.431 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.4 M 2024-04-16T14:59:25.431 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 29 M 2024-04-16T14:59:25.431 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:26.730 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-19.0.0-3100.g3f7db60e.el9.x86_64.rpm 2.6 MB/s | 3.4 MB 00:01 2024-04-16T14:59:26.731 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:26.731 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.6 MB/s | 3.4 MB 00:01 2024-04-16T14:59:26.731 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:26.745 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:26.745 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:26.824 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:26.824 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:27.386 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:27.440 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:27.465 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:27.465 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T14:59:27.466 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T14:59:27.466 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T14:59:27.466 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T14:59:27.466 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:28.782 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:28.782 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:28.782 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:28.782 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:28.782 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:28.782 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:29.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2024-04-16T14:59:29.663 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:51 ago on Tue 16 Apr 2024 02:54:38 PM UTC. 2024-04-16T14:59:29.817 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-3100.g3f7db60e.el9 ceph 175 k 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-16T14:59:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-04-16T14:59:29.819 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:29.819 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 175 k 2024-04-16T14:59:29.819 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 673 k 2024-04-16T14:59:29.819 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-16T14:59:30.260 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-19.0.0-3100.g3f7db60e.el9.x86_64.rpm 398 kB/s | 175 kB 00:00 2024-04-16T14:59:30.261 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-16T14:59:30.261 INFO:teuthology.orchestra.run.smithi097.stdout:Total 397 kB/s | 175 kB 00:00 2024-04-16T14:59:30.261 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-16T14:59:30.271 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-16T14:59:30.271 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-16T14:59:30.328 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-16T14:59:30.329 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-16T14:59:30.495 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:31.037 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:31.314 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 1/1 2024-04-16T14:59:31.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:31.314 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-16T14:59:31.314 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-3100.g3f7db60e.el9.x86_64 2024-04-16T14:59:31.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:31.314 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-16T14:59:31.563 DEBUG:teuthology.parallel:result is None 2024-04-16T14:59:31.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T14:59:31.709 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T14:59:31.731 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-3100.g3f7db60e.el9 2024-04-16T14:59:31.731 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3100.g3f7db60e.el9 2024-04-16T14:59:31.731 INFO:teuthology.task.install:The correct ceph version 19.0.0-3100.g3f7db60e is installed. 2024-04-16T14:59:31.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T14:59:31.877 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T14:59:31.903 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3100.g3f7db60e.el9 2024-04-16T14:59:31.903 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3100.g3f7db60e.el9 2024-04-16T14:59:31.904 INFO:teuthology.task.install:The correct ceph version 19.0.0-3100.g3f7db60e is installed. 2024-04-16T14:59:31.905 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T14:59:31.905 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:31.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T14:59:31.935 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:31.935 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T14:59:31.977 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T14:59:31.977 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:31.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T14:59:32.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T14:59:32.075 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:32.075 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T14:59:32.108 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T14:59:32.178 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T14:59:32.179 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:32.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T14:59:32.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T14:59:32.275 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:32.275 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T14:59:32.305 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T14:59:32.374 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T14:59:32.375 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:32.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T14:59:32.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T14:59:32.470 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:32.470 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T14:59:32.500 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T14:59:32.569 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T14:59:32.660 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T14:59:32.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2024-04-16T14:59:32.663 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-04-16T14:59:32.693 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T14:59:32.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T14:59:32.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T14:59:32.762 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T14:59:32.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T14:59:32.801 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T14:59:32.840 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T14:59:32.841 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T14:59:32.841 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391', 'branch': 'main', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7658391', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-thrash-osd', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '372751789be1725ed06fd077d029295c8cab4c35'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 3903, 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-thrash-osd', 'suite_path': '/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/Matan-B/ceph.git', 'suite_sha1': '372751789be1725ed06fd077d029295c8cab4c35', 'targets': {'smithi097.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNXmzu0aqrX/brCJad7hOOc5xrCmRP7WJ6BEU7E69d9quoupN6W7mtUvqmVhcN3YHewmLUVRlhtjINfAKQI77AI=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjG692fe51fvXOSbVE4fA2zMap3ZdmEsjONTdJZdF9fG3q77p0KwYF+U68bWtBouhDOJwkcwHw+f4VGRs5dCew='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_14:35:17', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T14:59:32.841 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T14:59:32.870 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T14:59:32.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T14:59:32.926 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T14:59:32.955 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:32.956 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T14:59:33.008 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T14:59:33.008 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-04-16T14:59:33.064 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T14:59:33.064 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:33.064 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-16T14:59:33.064 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:33.064 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:33.065 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-16 14:59:27.706145971 +0000 2024-04-16T14:59:33.065 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-16 14:53:47.456396768 +0000 2024-04-16T14:59:33.065 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-16 14:53:47.456396768 +0000 2024-04-16T14:59:33.065 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-16 14:53:47.456396768 +0000 2024-04-16T14:59:33.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T14:59:33.131 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-16T14:59:33.131 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-16T14:59:33.131 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000237952 s, 2.2 MB/s 2024-04-16T14:59:33.133 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T14:59:33.189 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-04-16T14:59:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T14:59:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-16 14:59:27.707145954 +0000 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-16 14:53:47.801390814 +0000 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-16 14:53:47.801390814 +0000 2024-04-16T14:59:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-16 14:53:47.801390814 +0000 2024-04-16T14:59:33.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T14:59:33.312 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-16T14:59:33.312 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-16T14:59:33.312 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000194316 s, 2.6 MB/s 2024-04-16T14:59:33.313 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T14:59:33.369 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-16 14:59:27.707145954 +0000 2024-04-16T14:59:33.426 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-16 14:53:48.146384860 +0000 2024-04-16T14:59:33.427 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-16 14:53:48.146384860 +0000 2024-04-16T14:59:33.427 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-16 14:53:48.146384860 +0000 2024-04-16T14:59:33.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T14:59:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-16T14:59:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-16T14:59:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000209999 s, 2.4 MB/s 2024-04-16T14:59:33.494 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T14:59:33.552 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-16 14:59:27.707145954 +0000 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-16 14:53:48.500378750 +0000 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-16 14:53:48.500378750 +0000 2024-04-16T14:59:33.608 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-16 14:53:48.500378750 +0000 2024-04-16T14:59:33.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T14:59:33.674 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-16T14:59:33.675 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-16T14:59:33.675 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000210597 s, 2.4 MB/s 2024-04-16T14:59:33.676 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T14:59:33.732 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T14:59:33.733 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:33.733 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T14:59:33.749 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T14:59:33.750 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-16T14:59:33.805 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T14:59:33.805 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-16 14:59:03.655096751 +0000 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-16 14:53:39.853699587 +0000 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-16 14:53:39.853699587 +0000 2024-04-16T14:59:33.806 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-16 14:53:39.853699587 +0000 2024-04-16T14:59:33.806 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T14:59:33.872 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-16T14:59:33.872 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-16T14:59:33.872 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000181069 s, 2.8 MB/s 2024-04-16T14:59:33.873 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T14:59:33.930 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-16 14:59:03.656096734 +0000 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-16 14:53:40.222693016 +0000 2024-04-16T14:59:33.986 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-16 14:53:40.222693016 +0000 2024-04-16T14:59:33.987 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-16 14:53:40.222693016 +0000 2024-04-16T14:59:33.987 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T14:59:34.053 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-16T14:59:34.053 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-16T14:59:34.053 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000236116 s, 2.2 MB/s 2024-04-16T14:59:34.054 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T14:59:34.111 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-16T14:59:34.166 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T14:59:34.166 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:34.166 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-16T14:59:34.166 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:34.166 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:34.167 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-16 14:59:03.656096734 +0000 2024-04-16T14:59:34.167 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-16 14:53:40.575686730 +0000 2024-04-16T14:59:34.167 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-16 14:53:40.575686730 +0000 2024-04-16T14:59:34.167 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-16 14:53:40.575686730 +0000 2024-04-16T14:59:34.167 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T14:59:34.233 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-16T14:59:34.233 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-16T14:59:34.233 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000216371 s, 2.4 MB/s 2024-04-16T14:59:34.234 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T14:59:34.290 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-16 14:59:03.657096716 +0000 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-16 14:53:41.084677667 +0000 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-16 14:53:41.084677667 +0000 2024-04-16T14:59:34.346 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-16 14:53:41.084677667 +0000 2024-04-16T14:59:34.346 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T14:59:34.413 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-16T14:59:34.413 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-16T14:59:34.413 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00021096 s, 2.4 MB/s 2024-04-16T14:59:34.414 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T14:59:34.470 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T14:59:34.471 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T14:59:34.471 INFO:tasks.ceph:Generating config... 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-16T14:59:34.472 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] log to file = False 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-16T14:59:34.473 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T14:59:34.474 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-16T14:59:34.474 INFO:tasks.ceph:[client.0] debug ms = 1 2024-04-16T14:59:34.474 INFO:tasks.ceph:[client.0] debug objecter = 20 2024-04-16T14:59:34.474 INFO:tasks.ceph:[client.0] debug rados = 20 2024-04-16T14:59:34.474 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T14:59:34.474 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T14:59:34.474 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T14:59:34.474 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T14:59:34.475 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T14:59:34.475 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T14:59:34.475 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T14:59:34.475 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-16T14:59:34.475 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T14:59:34.475 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T14:59:34.476 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T14:59:34.476 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T14:59:34.476 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-16T14:59:34.476 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-16T14:59:34.476 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-16T14:59:34.476 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-16T14:59:34.476 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T14:59:34.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T14:59:34.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T14:59:34.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T14:59:34.634 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.97')] 2024-04-16T14:59:34.635 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.97', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-16T14:59:34.636 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:34.636 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T14:59:34.691 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.97 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:34.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T14:59:34.782+0000 7f31cdc65f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T14:59:34.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T14:59:34.782+0000 7f31cdc65f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T14:59:34.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T14:59:34.783+0000 7f31cdc65f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: generated fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = quincy 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 0 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:last_changed 2024-04-16T14:59:34.784108+0000 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-04-16T14:59:34.784108+0000 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:min_mon_release 17 (quincy) 2024-04-16T14:59:34.782 INFO:teuthology.orchestra.run.smithi097.stdout:election_strategy: 1 2024-04-16T14:59:34.783 INFO:teuthology.orchestra.run.smithi097.stdout:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-16T14:59:34.783 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T14:59:34.784 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T14:59:34.799 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cb83d412-954b-4b45-966f-51f0bfc0a486... 2024-04-16T14:59:34.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T14:59:34.841 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.892 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-04-16T14:59:34.893 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.894 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = 172.21.15.97 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: ms client mode = crc 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: ms cluster mode = crc 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: ms mon client mode = crc 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: ms mon cluster mode = crc 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: ms mon service mode = crc 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: ms service mode = crc 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_min_size = 2 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_size = 3 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:34.895 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 120 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-04-16T14:59:34.896 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.897 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: debug monc = 20 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore compression algorithm = snappy 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore compression mode = aggressive 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore rocksdb cf = False 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: crimson osd obc lru size = 10 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 20 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 20 2024-04-16T14:59:34.898 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 10 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: err to stderr = True 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: log flush on exit = True 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: log to file = False 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: log to stderr = True 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd blocked scrub grace period = 3600 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug reject backfill probability = 0.3 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd delete sleep = 1 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd max backfills = 3 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd max markdown count = 1000 2024-04-16T14:59:34.899 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub min interval = 60 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: osd shutdown pgref assert = True 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: osd snap trim sleep = 2 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-04-16T14:59:34.900 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T14:59:34.901 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon min osdmap epochs = 50 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune interval = 2 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune min = 15 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune txsize = 2 2024-04-16T14:59:34.902 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default crimson = True 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: paxos service trim min = 10 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: debug objecter = 20 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: debug rados = 20 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout: crimson seastar cpu cores = 0-2 2024-04-16T14:59:34.903 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi097.stdout: crimson seastar cpu cores = 3-5 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi097.stdout: crimson seastar cpu cores = 0-2 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi097.stdout: crimson seastar cpu cores = 3-5 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-04-16T14:59:34.904 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-04-16T14:59:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-04-16T14:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-04-16T14:59:34.907 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = 172.21.15.97 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: ms client mode = crc 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: ms cluster mode = crc 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: ms mon client mode = crc 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: ms mon cluster mode = crc 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: ms mon service mode = crc 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: ms service mode = crc 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_min_size = 2 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_size = 3 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-04-16T14:59:34.908 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 120 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-04-16T14:59:34.909 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-04-16T14:59:34.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore compression algorithm = snappy 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore compression mode = aggressive 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore rocksdb cf = False 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: crimson osd obc lru size = 10 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 20 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 20 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-04-16T14:59:34.911 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 10 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: err to stderr = True 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: log flush on exit = True 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: log to file = False 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: log to stderr = True 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd blocked scrub grace period = 3600 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug reject backfill probability = 0.3 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd delete sleep = 1 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd max backfills = 3 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd max markdown count = 1000 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub min interval = 60 2024-04-16T14:59:34.912 INFO:teuthology.orchestra.run.smithi186.stdout: osd shutdown pgref assert = True 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: osd snap trim sleep = 2 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-16T14:59:34.913 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T14:59:34.914 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon min osdmap epochs = 50 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune interval = 2 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune min = 15 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune txsize = 2 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default crimson = True 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: paxos service trim min = 10 2024-04-16T14:59:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: debug objecter = 20 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: debug rados = 20 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 0-2 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 3-5 2024-04-16T14:59:34.916 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-04-16T14:59:34.917 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 0-2 2024-04-16T14:59:34.917 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-04-16T14:59:34.917 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 3-5 2024-04-16T14:59:34.918 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T14:59:34.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T14:59:34.985 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T14:59:34.985 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:34.986 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T14:59:35.001 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:35.001 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T14:59:35.057 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T14:59:35.057 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:35.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T14:59:35.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T14:59:35.139 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T14:59:35.139 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:35.194 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T14:59:35.195 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:35.195 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T14:59:35.195 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T14:59:35.236 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T14:59:35.236 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:35.292 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T14:59:35.293 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T14:59:35.293 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T14:59:35.405 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T14:59:35.408 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T14:59:35.408 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T14:59:35.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T14:59:35.460 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T14:59:35.476 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T14:59:35.529 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-16T14:59:35.546 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T14:59:35.546 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T14:59:35.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T14:59:35.574 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T14:59:35.574 INFO:tasks.ceph:role: osd.0 2024-04-16T14:59:35.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T14:59:35.575 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T14:59:35.650 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T14:59:35.651 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T14:59:36.159 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-04-16T14:59:36.161 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-04-16T14:59:36.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T14:59:36.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T14:59:36.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T14:59:36.420 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T14:59:36.420 INFO:tasks.ceph:role: osd.1 2024-04-16T14:59:36.420 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T14:59:36.420 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T14:59:36.497 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T14:59:36.498 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T14:59:36.498 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T14:59:36.943 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-04-16T14:59:36.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-04-16T14:59:36.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T14:59:37.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T14:59:37.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:37.417 INFO:teuthology.orchestra.run.smithi097.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:37,421 seastar - Reactor backend: linux-aio 2024-04-16T14:59:37.430 INFO:teuthology.orchestra.run.smithi097.stderr:==29038==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:37.558 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:37,560 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T14:59:37.558 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:37,560 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T14:59:37.558 INFO:teuthology.orchestra.run.smithi097.stderr:DEBUG 2024-04-16 14:59:37,561 [shard 0:main] seastar - reactor::drain 2024-04-16T14:59:38.125 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:38.126 INFO:teuthology.orchestra.run.smithi097.stderr:Suppressions used: 2024-04-16T14:59:38.126 INFO:teuthology.orchestra.run.smithi097.stderr: count bytes template 2024-04-16T14:59:38.126 INFO:teuthology.orchestra.run.smithi097.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:38.126 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:38.126 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-16T14:59:38.181 INFO:teuthology.orchestra.run.smithi097.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:38.184 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:38,187 seastar - Reactor backend: linux-aio 2024-04-16T14:59:38.193 INFO:teuthology.orchestra.run.smithi097.stderr:==29036==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:42.833 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:42.834 INFO:teuthology.orchestra.run.smithi097.stderr:Suppressions used: 2024-04-16T14:59:42.834 INFO:teuthology.orchestra.run.smithi097.stderr: count bytes template 2024-04-16T14:59:42.834 INFO:teuthology.orchestra.run.smithi097.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:42.834 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:42.834 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-16T14:59:42.834 INFO:teuthology.orchestra.run.smithi097.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:42.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T14:59:42.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:43.182 INFO:teuthology.orchestra.run.smithi097.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:43.183 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:43,185 seastar - Reactor backend: linux-aio 2024-04-16T14:59:43.188 INFO:teuthology.orchestra.run.smithi097.stderr:==29221==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi097.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb997509 0xb869fd0 0xb86b54c 0xb86b762 0xb86b8b8 0xb86bd81 0x54daf 0x172ee 0x174bb 0x2eab 0x4a09d 0x35b7c 0x36eee 0x5daa973 0x5daace4 0x5dab0c6 0x5e07054 0x5d85bd0 0x5d86104 0x5bcadf9 0x5be5ebd 0x6ec0764 0x6ed1f37 0x6ed33d4 0x6ed3459 0x6ed355b 0x6ed3653 0x6ed3860 0x6ed3a46 0x6ed3d95 0x6ed4139 0x6ed4369 0x6ed45d3 0x6ed48a0 0x6ed4abe 0x6ed4b76 0x6ed4c3c 0x6ed4d1d 0x5a336f8 0x5a33802 0x5a338be 0x5a3397a 0x5a33a36 0x5a33b83 0x5a33d0e 0x5a340cf 0x5a50010 0xb87cc55 0xb896eef 0xb937eed 0xb93953b 0xb615d21 0xb61669d 0x37d5b74 0x37d66cd 0x3675d6a 0x3feaf 0x3ff5f 0x3462434 2024-04-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi097.stderr:kernel callstack: 2024-04-16T14:59:43.312 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:43,314 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T14:59:43.312 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:43,315 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T14:59:43.313 INFO:teuthology.orchestra.run.smithi097.stderr:DEBUG 2024-04-16 14:59:43,315 [shard 0:main] seastar - reactor::drain 2024-04-16T14:59:43.851 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:43.851 INFO:teuthology.orchestra.run.smithi097.stderr:Suppressions used: 2024-04-16T14:59:43.851 INFO:teuthology.orchestra.run.smithi097.stderr: count bytes template 2024-04-16T14:59:43.851 INFO:teuthology.orchestra.run.smithi097.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:43.851 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:43.851 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-16T14:59:43.913 INFO:teuthology.orchestra.run.smithi097.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:43.916 INFO:teuthology.orchestra.run.smithi097.stderr:INFO 2024-04-16 14:59:43,919 seastar - Reactor backend: linux-aio 2024-04-16T14:59:43.935 INFO:teuthology.orchestra.run.smithi097.stderr:==29219==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:48.600 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:48.600 INFO:teuthology.orchestra.run.smithi097.stderr:Suppressions used: 2024-04-16T14:59:48.600 INFO:teuthology.orchestra.run.smithi097.stderr: count bytes template 2024-04-16T14:59:48.600 INFO:teuthology.orchestra.run.smithi097.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:48.600 INFO:teuthology.orchestra.run.smithi097.stderr:----------------------------------------------------- 2024-04-16T14:59:48.600 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-16T14:59:48.601 INFO:teuthology.orchestra.run.smithi097.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:48.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T14:59:48.650 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T14:59:48.684 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T14:59:48.684 INFO:tasks.ceph:role: osd.2 2024-04-16T14:59:48.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T14:59:48.684 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T14:59:48.764 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T14:59:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T14:59:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T14:59:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T14:59:48.764 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T14:59:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-16T14:59:48.765 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T14:59:48.765 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T14:59:48.765 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T14:59:48.765 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T14:59:49.256 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-16T14:59:49.259 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-16T14:59:49.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-16T14:59:49.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T14:59:49.437 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-16T14:59:49.509 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T14:59:49.509 INFO:tasks.ceph:role: osd.3 2024-04-16T14:59:49.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T14:59:49.509 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T14:59:49.587 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T14:59:50.056 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-16T14:59:50.059 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-16T14:59:50.059 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-16T14:59:50.154 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-16T14:59:50.189 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:50.526 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:50.527 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:50,530 seastar - Reactor backend: linux-aio 2024-04-16T14:59:50.538 INFO:teuthology.orchestra.run.smithi186.stderr:==29356==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:50,665 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T14:59:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:50,665 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T14:59:50.664 INFO:teuthology.orchestra.run.smithi186.stderr:DEBUG 2024-04-16 14:59:50,666 [shard 0:main] seastar - reactor::drain 2024-04-16T14:59:51.221 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T14:59:51.221 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-16T14:59:51.221 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-16T14:59:51.221 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:51.221 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T14:59:51.221 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-16T14:59:51.280 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:51.284 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:51,286 seastar - Reactor backend: linux-aio 2024-04-16T14:59:51.293 INFO:teuthology.orchestra.run.smithi186.stderr:==29354==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-16T14:59:55.925 INFO:teuthology.orchestra.run.smithi186.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T14:59:55.949 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T14:59:55.977 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T14:59:56.275 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:56.276 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:56,279 seastar - Reactor backend: linux-aio 2024-04-16T14:59:56.281 INFO:teuthology.orchestra.run.smithi186.stderr:==29538==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T14:59:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb997509 0xb869fd0 0xb86b54c 0xb86b762 0xb86b8b8 0xb86bd81 0x54daf 0x349b9d2 0x5d829a3 0x5d85e34 0x5d86336 0x5bcadf9 0x5be5ebd 0x6ec0764 0x6ed1f37 0x6ed33d4 0x6ed3459 0x6ed355b 0x6ed3653 0x6ed3860 0x6ed3a46 0x6ed3d95 0x6ed4139 0x6ed4369 0x6ed45d3 0x6ed48a0 0x6ed4abe 0x6ed4b76 0x6ed4c3c 0x6ed4d1d 0x5a336f8 0x5a33802 0x5a338be 0x5a3397a 0x5a33a36 0x5a33b83 0x5a33d0e 0x5a340cf 0x5a50010 0xb87cc55 0xb896eef 0xb937eed 0xb93953b 0xb615d21 0xb61669d 0x37d5b74 0x37d66cd 0x3675d6a 0x3feaf 0x3ff5f 0x3462434 2024-04-16T14:59:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:kernel callstack: 2024-04-16T14:59:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:56,410 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T14:59:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:56,410 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T14:59:56.408 INFO:teuthology.orchestra.run.smithi186.stderr:DEBUG 2024-04-16 14:59:56,411 [shard 0:main] seastar - reactor::drain 2024-04-16T14:59:56.946 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T14:59:56.947 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-16T14:59:56.947 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-16T14:59:56.947 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-16T14:59:56.947 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T14:59:56.947 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-16T14:59:57.004 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T14:59:57.008 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-16 14:59:57,010 seastar - Reactor backend: linux-aio 2024-04-16T14:59:57.022 INFO:teuthology.orchestra.run.smithi186.stderr:==29536==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-16T15:00:01.690 INFO:teuthology.orchestra.run.smithi186.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T15:00:01.713 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T15:00:01.742 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T15:00:01.742 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T15:00:01.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T15:00:01.777 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T15:00:01.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T15:00:01.849 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T15:00:01.850 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T15:00:01.878 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T15:00:01.878 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T15:00:01.947 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T15:00:01.947 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-16T15:00:02.016 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T15:00:02.016 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T15:00:02.035 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T15:00:02.035 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-16T15:00:02.072 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T15:00:02.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBakh5mSMyCFBAAhsBTeTE7PwLYp+h/XuGIhw== 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBgkh5mWzEkBRAA1f2t9kxyKj50ms+6f5alHA== 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.x] 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBXkh5mOQFLGBAABR+gqHFQBnKTaYxk2m22eQ== 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2024-04-16T15:00:02.106 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBnkh5m9hipGhAArl9e2OV7MZ/RTlX7l6tyow== 2024-04-16T15:00:02.107 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2024-04-16T15:00:02.107 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBtkh5mKzaWChAAeQS/qYc0TwCCo6zUxwIJPQ== 2024-04-16T15:00:02.107 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-04-16T15:00:02.107 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBXkh5mbUGeGxAAxuOAQrH8eZVeJjzjBPHi2w== 2024-04-16T15:00:02.107 INFO:teuthology.orchestra.run.smithi097.stdout:[client.1] 2024-04-16T15:00:02.107 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBXkh5mG4a+HxAA6WGUCNrfLDpLHOze+Q0LOA== 2024-04-16T15:00:02.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T15:00:02.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T15:00:02.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T15:00:02.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T15:00:02.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T15:00:02.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T15:00:02.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T15:00:02.611 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T15:00:02.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T15:00:02.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T15:00:02.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:02.763+0000 7f98e945da80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:02.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:02.763+0000 7f98e945da80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:02.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T15:00:02.803 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T15:00:02.859 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T15:00:02.859 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T15:00:02.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T15:00:02.901 INFO:tasks.ceph.mon.a:Started 2024-04-16T15:00:02.902 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T15:00:02.902 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T15:00:02.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T15:00:02.907 INFO:tasks.ceph.mgr.x:Started 2024-04-16T15:00:02.907 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T15:00:02.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T15:00:02.988 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-16T15:00:02.989+0000 7fe536265a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:02.988 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-16T15:00:02.989+0000 7fe536265a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:02.988 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-16T15:00:02.990+0000 7fe536265a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.007 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:03.009+0000 7f2491168000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:03.016+0000 7fe73b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.022 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:03.024+0000 7f2491168000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.022 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:03.024+0000 7f2491168000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:03.026+0000 7fe73b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.095 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-04-16T15:00:03.109 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T15:00:03.109 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T15:00:03.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T15:00:03.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:03.216+0000 7f713299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:03.225+0000 7f713299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:03.303 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-16T15:00:03.305+0000 7fe52cff4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T15:00:03.303 INFO:tasks.ceph.mon.a.smithi097.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T15:00:03.303 INFO:tasks.ceph.mon.a.smithi097.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T15:00:03.303 INFO:tasks.ceph.mon.a.smithi097.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T15:00:03.303 INFO:tasks.ceph.mon.a.smithi097.stderr:feature with important data. 2024-04-16T15:00:03.303 INFO:tasks.ceph.mon.a.smithi097.stderr: 2024-04-16T15:00:03.482 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:03.484+0000 7f2491168000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T15:00:03.536 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:03.538+0000 7f2491168000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T15:00:04.065 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.066+0000 7f2491168000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T15:00:04.111 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T15:00:04.111 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T15:00:04.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T15:00:04.118 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.120+0000 7f2491168000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T15:00:04.143 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-16T15:00:04.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T15:00:04.163 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.165+0000 7f2491168000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T15:00:04.213 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T15:00:04.214 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T15:00:04.217 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.219+0000 7f2491168000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T15:00:04.242 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-16T15:00:04.242 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-16T15:00:04.262 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.264+0000 7f2491168000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T15:00:04.290 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 125ac5fe-95d0-4d08-8c00-e7aba9a62e13 0 2024-04-16T15:00:04.386 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.388+0000 7f2491168000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T15:00:04.443 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:04.439+0000 7fb28aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:04.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:04.448+0000 7fb28aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:04.462 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.463+0000 7f2491168000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T15:00:04.510 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.511+0000 7f2491168000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T15:00:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-16T15:00:04.538 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new e17fd5ff-0cc5-4287-aca6-1e8916306c65 1 2024-04-16T15:00:04.566 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.568+0000 7f2491168000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T15:00:04.616 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:04.618+0000 7f2491168000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T15:00:04.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:04.645+0000 7f776035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:04.654+0000 7f776035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:04.732 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-04-16T15:00:04.744 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 8b186edc-9c46-4545-9f0f-90a8a3372477 2 2024-04-16T15:00:04.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:04.895+0000 7f38ce59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:04.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:04.904+0000 7f38ce59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-04-16T15:00:04.995 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 46950121-5123-4075-a5fe-dd05b8c4a353 3 2024-04-16T15:00:05.018 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.019+0000 7f2491168000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T15:00:05.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:05.104+0000 7f94ef26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:05.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-16T15:00:05.113+0000 7f94ef26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:05.153 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.155+0000 7f2491168000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T15:00:05.192 INFO:teuthology.orchestra.run.smithi186.stdout:3 2024-04-16T15:00:05.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T15:00:05.243 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T15:00:05.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T15:00:05.285 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.286+0000 7f2491168000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T15:00:05.286 INFO:tasks.ceph.osd.0:Started 2024-04-16T15:00:05.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T15:00:05.321 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T15:00:05.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T15:00:05.330 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.332+0000 7f2491168000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T15:00:05.363 INFO:tasks.ceph.osd.1:Started 2024-04-16T15:00:05.365 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T15:00:05.399 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T15:00:05.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T15:00:05.441 INFO:tasks.ceph.osd.2:Started 2024-04-16T15:00:05.442 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-16T15:00:05.446 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.448+0000 7f2491168000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T15:00:05.474 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T15:00:05.474 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T15:00:05.517 INFO:tasks.ceph.osd.3:Started 2024-04-16T15:00:05.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T15:00:05.551 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.552+0000 7f2491168000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T15:00:05.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:05.635+0000 7fe66899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:05.643+0000 7fe66899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":7,"fsid":"cb83d412-954b-4b45-966f-51f0bfc0a486","created":"2024-04-16T15:00:03.012552+0000","modified":"2024-04-16T15:00:05.193619+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T15:00:05.193619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"125ac5fe-95d0-4d08-8c00-e7aba9a62e13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e17fd5ff-0cc5-4287-aca6-1e8916306c65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b186edc-9c46-4545-9f0f-90a8a3372477","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46950121-5123-4075-a5fe-dd05b8c4a353","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T15:00:05.721 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T15:00:05.721 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T15:00:05.726 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.728+0000 7f2491168000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T15:00:05.778 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.778+0000 7f2491168000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T15:00:05.943 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:05.944+0000 7f2491168000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T15:00:06.124 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T15:00:06.229 INFO:teuthology.misc.health.smithi097.stderr:2024-04-16T15:00:06.227+0000 7fb90ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:06.233 INFO:teuthology.misc.health.smithi097.stderr:2024-04-16T15:00:06.235+0000 7fb90ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:06.303 INFO:teuthology.misc.health.smithi097.stdout: 2024-04-16T15:00:06.303 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":7,"fsid":"cb83d412-954b-4b45-966f-51f0bfc0a486","created":"2024-04-16T15:00:03.012552+0000","modified":"2024-04-16T15:00:05.193619+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T15:00:05.193619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"125ac5fe-95d0-4d08-8c00-e7aba9a62e13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e17fd5ff-0cc5-4287-aca6-1e8916306c65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b186edc-9c46-4545-9f0f-90a8a3372477","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46950121-5123-4075-a5fe-dd05b8c4a353","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T15:00:06.313 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-16T15:00:06.835 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:06.836+0000 7f2491168000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T15:00:07.124 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:07.124+0000 7f2491168000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T15:00:07.179 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:07.180+0000 7f2491168000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T15:00:07.311 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:07.312+0000 7f2491168000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T15:00:08.322 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:00:08.323+0000 7f24302dc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T15:00:12.717 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T15:00:12.829 INFO:teuthology.misc.health.smithi097.stderr:2024-04-16T15:00:12.827+0000 7fd6f942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:12.833 INFO:teuthology.misc.health.smithi097.stderr:2024-04-16T15:00:12.835+0000 7fd6f942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:13.022 INFO:teuthology.misc.health.smithi097.stdout: 2024-04-16T15:00:13.022 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"cb83d412-954b-4b45-966f-51f0bfc0a486","created":"2024-04-16T15:00:03.012552+0000","modified":"2024-04-16T15:00:12.336043+0000","last_up_change":"2024-04-16T15:00:10.325547+0000","last_in_change":"2024-04-16T15:00:05.193619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T15:00:11.342351+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"125ac5fe-95d0-4d08-8c00-e7aba9a62e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2701876556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":2701876556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2701876556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6803","nonce":2701876556}]},"public_addr":"172.21.15.97:6804/2701876556","cluster_addr":"172.21.15.97:6805/2701876556","heartbeat_back_addr":"172.21.15.97:6806/2701876556","heartbeat_front_addr":"172.21.15.97:6803/2701876556","state":["exists","up"]},{"osd":1,"uuid":"e17fd5ff-0cc5-4287-aca6-1e8916306c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6801","nonce":2873842391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2873842391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":2873842391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2873842391}]},"public_addr":"172.21.15.97:6801/2873842391","cluster_addr":"172.21.15.97:6802/2873842391","heartbeat_back_addr":"172.21.15.97:6809/2873842391","heartbeat_front_addr":"172.21.15.97:6808/2873842391","state":["exists","up"]},{"osd":2,"uuid":"8b186edc-9c46-4545-9f0f-90a8a3372477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4142591846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6805","nonce":4142591846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":4142591846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":4142591846}]},"public_addr":"172.21.15.186:6804/4142591846","cluster_addr":"172.21.15.186:6805/4142591846","heartbeat_back_addr":"172.21.15.186:6816/4142591846","heartbeat_front_addr":"172.21.15.186:6815/4142591846","state":["exists","up"]},{"osd":3,"uuid":"46950121-5123-4075-a5fe-dd05b8c4a353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":702210044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":702210044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":702210044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6803","nonce":702210044}]},"public_addr":"172.21.15.186:6808/702210044","cluster_addr":"172.21.15.186:6809/702210044","heartbeat_back_addr":"172.21.15.186:6806/702210044","heartbeat_front_addr":"172.21.15.186:6803/702210044","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T15:00:13.033 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-16T15:00:13.034 INFO:tasks.ceph:Creating RBD pool 2024-04-16T15:00:13.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T15:00:13.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:13.142+0000 7f7462b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:13.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:13.152+0000 7f7462b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:14.343 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'rbd' created 2024-04-16T15:00:14.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T15:00:14.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:14.484+0000 7f59abb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:14.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:14.506+0000 7f59abb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:15.344 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T15:00:15.361 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T15:00:15.361 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T15:00:15.362 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T15:00:15.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T15:00:15.362 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T15:00:15.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:15.518+0000 7f1fff19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:15.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:15.526+0000 7f1fff19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:15.764 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:15.775 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":67941034},{"type":"v1","addr":"172.21.15.186:6801","nonce":67941034}]},"active_addr":"172.21.15.186:6801/67941034","active_change":"2024-04-16T15:00:07.315765+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":449197765}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3618339828}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3279803312}]}]} 2024-04-16T15:00:15.780 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T15:00:15.780 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T15:00:15.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T15:00:15.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:15.889+0000 7f6290e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:15.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:15.896+0000 7f6290e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":13,"fsid":"cb83d412-954b-4b45-966f-51f0bfc0a486","created":"2024-04-16T15:00:03.012552+0000","modified":"2024-04-16T15:00:15.346059+0000","last_up_change":"2024-04-16T15:00:10.325547+0000","last_in_change":"2024-04-16T15:00:05.193619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T15:00:11.342351+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T15:00:13.350944+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"125ac5fe-95d0-4d08-8c00-e7aba9a62e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2701876556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":2701876556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2701876556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6803","nonce":2701876556}]},"public_addr":"172.21.15.97:6804/2701876556","cluster_addr":"172.21.15.97:6805/2701876556","heartbeat_back_addr":"172.21.15.97:6806/2701876556","heartbeat_front_addr":"172.21.15.97:6803/2701876556","state":["exists","up"]},{"osd":1,"uuid":"e17fd5ff-0cc5-4287-aca6-1e8916306c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6801","nonce":2873842391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2873842391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":2873842391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2873842391}]},"public_addr":"172.21.15.97:6801/2873842391","cluster_addr":"172.21.15.97:6802/2873842391","heartbeat_back_addr":"172.21.15.97:6809/2873842391","heartbeat_front_addr":"172.21.15.97:6808/2873842391","state":["exists","up"]},{"osd":2,"uuid":"8b186edc-9c46-4545-9f0f-90a8a3372477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4142591846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6805","nonce":4142591846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":4142591846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":4142591846}]},"public_addr":"172.21.15.186:6804/4142591846","cluster_addr":"172.21.15.186:6805/4142591846","heartbeat_back_addr":"172.21.15.186:6816/4142591846","heartbeat_front_addr":"172.21.15.186:6815/4142591846","state":["exists","up"]},{"osd":3,"uuid":"46950121-5123-4075-a5fe-dd05b8c4a353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":702210044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":702210044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":702210044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6803","nonce":702210044}]},"public_addr":"172.21.15.186:6808/702210044","cluster_addr":"172.21.15.186:6809/702210044","heartbeat_back_addr":"172.21.15.186:6806/702210044","heartbeat_front_addr":"172.21.15.186:6803/702210044","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T15:00:16.097 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T15:00:16.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T15:00:16.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.207+0000 7f5656206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.215+0000 7f5656206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:16.404 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":14,"fsid":"cb83d412-954b-4b45-966f-51f0bfc0a486","created":"2024-04-16T15:00:03.012552+0000","modified":"2024-04-16T15:00:16.358359+0000","last_up_change":"2024-04-16T15:00:10.325547+0000","last_in_change":"2024-04-16T15:00:05.193619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T15:00:11.342351+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T15:00:13.350944+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"125ac5fe-95d0-4d08-8c00-e7aba9a62e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2701876556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":2701876556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2701876556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6803","nonce":2701876556}]},"public_addr":"172.21.15.97:6804/2701876556","cluster_addr":"172.21.15.97:6805/2701876556","heartbeat_back_addr":"172.21.15.97:6806/2701876556","heartbeat_front_addr":"172.21.15.97:6803/2701876556","state":["exists","up"]},{"osd":1,"uuid":"e17fd5ff-0cc5-4287-aca6-1e8916306c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6801","nonce":2873842391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2873842391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":2873842391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2873842391}]},"public_addr":"172.21.15.97:6801/2873842391","cluster_addr":"172.21.15.97:6802/2873842391","heartbeat_back_addr":"172.21.15.97:6809/2873842391","heartbeat_front_addr":"172.21.15.97:6808/2873842391","state":["exists","up"]},{"osd":2,"uuid":"8b186edc-9c46-4545-9f0f-90a8a3372477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4142591846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6805","nonce":4142591846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":4142591846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":4142591846}]},"public_addr":"172.21.15.186:6804/4142591846","cluster_addr":"172.21.15.186:6805/4142591846","heartbeat_back_addr":"172.21.15.186:6816/4142591846","heartbeat_front_addr":"172.21.15.186:6815/4142591846","state":["exists","up"]},{"osd":3,"uuid":"46950121-5123-4075-a5fe-dd05b8c4a353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":702210044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":702210044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":702210044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6803","nonce":702210044}]},"public_addr":"172.21.15.186:6808/702210044","cluster_addr":"172.21.15.186:6809/702210044","heartbeat_back_addr":"172.21.15.186:6806/702210044","heartbeat_front_addr":"172.21.15.186:6803/702210044","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T15:00:16.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T15:00:16.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T15:00:16.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T15:00:16.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-16T15:00:16.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.538+0000 7feebbcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.538+0000 7fefc906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.538+0000 7f026cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.538+0000 7f98f955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.548+0000 7f026cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.548+0000 7feebbcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.549+0000 7fefc906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.549+0000 7f98f955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.584 INFO:teuthology.orchestra.run.smithi097.stdout:38654705670 2024-04-16T15:00:16.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T15:00:16.586 INFO:teuthology.orchestra.run.smithi097.stdout:38654705670 2024-04-16T15:00:16.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T15:00:16.587 INFO:teuthology.orchestra.run.smithi097.stdout:38654705670 2024-04-16T15:00:16.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T15:00:16.588 INFO:teuthology.orchestra.run.smithi097.stdout:38654705670 2024-04-16T15:00:16.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T15:00:16.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.746+0000 7f661e59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.746+0000 7f66b5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.746+0000 7f5cd922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.746+0000 7f103895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.756+0000 7f5cd922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.756+0000 7f103895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.756+0000 7f661e59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:16.756+0000 7f66b5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:16.956 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:16.962 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:16.968 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:16.971 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T15:00:16.971 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:16.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T15:00:16.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T15:00:16.986 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T15:00:17.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T15:00:17.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T15:00:17.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T15:00:17.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T15:00:18.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.096+0000 7f0c0e606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.110+0000 7f0c0e606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.138+0000 7fe2f722d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.139+0000 7f73d0c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.148+0000 7fe2f722d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.143+0000 7ff3c586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.150+0000 7f73d0c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:18.152+0000 7ff3c586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:18.314 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:18.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T15:00:18.348 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:18.359 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:18.361 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T15:00:18.368 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:18.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T15:00:18.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T15:00:19.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T15:00:19.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T15:00:19.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T15:00:19.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T15:00:19.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.443+0000 7f6bba606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.454+0000 7f6bba606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.489+0000 7f69e66ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.499+0000 7f69e66ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.536+0000 7f4b285fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.545+0000 7f4b285fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.547+0000 7ff0f459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:19.557+0000 7ff0f459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:19.666 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:19.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T15:00:19.730 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:19.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T15:00:19.748 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:19.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T15:00:19.773 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:19.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T15:00:20.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T15:00:20.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T15:00:20.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T15:00:20.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T15:00:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.795+0000 7f081906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.805+0000 7f081906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.864+0000 7f67538ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.877+0000 7f67538ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.903+0000 7fed2b46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.904+0000 7f6702a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.913+0000 7fed2b46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:20.914+0000 7f6702a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:21.044 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:21.061 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T15:00:21.092 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:21.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T15:00:21.125 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:21.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T15:00:21.147 INFO:teuthology.orchestra.run.smithi097.stdout:38654705669 2024-04-16T15:00:21.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T15:00:22.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T15:00:22.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T15:00:22.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T15:00:22.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T15:00:22.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.179+0000 7f659422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.190+0000 7f659422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.226+0000 7fc3d9f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.236+0000 7fc3d9f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.276+0000 7f2988f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.286+0000 7f2988f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.300+0000 7f55816ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.309+0000 7f55816ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.402 INFO:teuthology.orchestra.run.smithi097.stdout:38654705674 2024-04-16T15:00:22.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.1 2024-04-16T15:00:22.416 DEBUG:teuthology.parallel:result is None 2024-04-16T15:00:22.504 INFO:teuthology.orchestra.run.smithi097.stdout:38654705674 2024-04-16T15:00:22.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.3 2024-04-16T15:00:22.517 DEBUG:teuthology.parallel:result is None 2024-04-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi097.stdout:38654705674 2024-04-16T15:00:22.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.0 2024-04-16T15:00:22.543 DEBUG:teuthology.parallel:result is None 2024-04-16T15:00:22.569 INFO:teuthology.orchestra.run.smithi097.stdout:38654705674 2024-04-16T15:00:22.582 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.2 2024-04-16T15:00:22.582 DEBUG:teuthology.parallel:result is None 2024-04-16T15:00:22.582 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T15:00:22.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T15:00:22.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.733+0000 7fbb222ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:22.740+0000 7fbb222ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-16T15:00:22.937 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-16T15:00:21.323525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89892,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.966401"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.455681+0000","last_change":"2024-04-16T15:00:15.457441+0000","last_active":"2024-04-16T15:00:15.455681+0000","last_peered":"2024-04-16T15:00:15.455681+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.444424+0000","last_became_peered":"2024-04-16T15:00:15.444424+0000","last_unstale":"2024-04-16T15:00:15.455681+0000","last_undegraded":"2024-04-16T15:00:15.455681+0000","last_fullsized":"2024-04-16T15:00:15.455681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.445837+0000","last_change":"2024-04-16T15:00:15.447176+0000","last_active":"2024-04-16T15:00:15.445837+0000","last_peered":"2024-04-16T15:00:15.445837+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.441817+0000","last_became_peered":"2024-04-16T15:00:15.441817+0000","last_unstale":"2024-04-16T15:00:15.445837+0000","last_undegraded":"2024-04-16T15:00:15.445837+0000","last_fullsized":"2024-04-16T15:00:15.445837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.375283+0000","last_change":"2024-04-16T15:00:15.451779+0000","last_active":"2024-04-16T15:00:16.375283+0000","last_peered":"2024-04-16T15:00:16.375283+0000","last_clean":"2024-04-16T15:00:16.375283+0000","last_became_active":"2024-04-16T15:00:15.444660+0000","last_became_peered":"2024-04-16T15:00:15.444660+0000","last_unstale":"2024-04-16T15:00:16.375283+0000","last_undegraded":"2024-04-16T15:00:16.375283+0000","last_fullsized":"2024-04-16T15:00:16.375283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.439719+0000","last_change":"2024-04-16T15:00:15.440602+0000","last_active":"2024-04-16T15:00:15.439719+0000","last_peered":"2024-04-16T15:00:15.439719+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.436185+0000","last_became_peered":"2024-04-16T15:00:15.436185+0000","last_unstale":"2024-04-16T15:00:15.439719+0000","last_undegraded":"2024-04-16T15:00:15.439719+0000","last_fullsized":"2024-04-16T15:00:15.439719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.452396+0000","last_change":"2024-04-16T15:00:15.453913+0000","last_active":"2024-04-16T15:00:15.452396+0000","last_peered":"2024-04-16T15:00:15.452396+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.447182+0000","last_became_peered":"2024-04-16T15:00:15.447182+0000","last_unstale":"2024-04-16T15:00:15.452396+0000","last_undegraded":"2024-04-16T15:00:15.452396+0000","last_fullsized":"2024-04-16T15:00:15.452396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.425720+0000","last_change":"2024-04-16T15:00:15.427043+0000","last_active":"2024-04-16T15:00:15.425720+0000","last_peered":"2024-04-16T15:00:15.425720+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.421570+0000","last_became_peered":"2024-04-16T15:00:15.421570+0000","last_unstale":"2024-04-16T15:00:15.425720+0000","last_undegraded":"2024-04-16T15:00:15.425720+0000","last_fullsized":"2024-04-16T15:00:15.425720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.466211+0000","last_change":"2024-04-16T15:00:15.468018+0000","last_active":"2024-04-16T15:00:15.466211+0000","last_peered":"2024-04-16T15:00:15.466211+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.457433+0000","last_became_peered":"2024-04-16T15:00:15.457433+0000","last_unstale":"2024-04-16T15:00:15.466211+0000","last_undegraded":"2024-04-16T15:00:15.466211+0000","last_fullsized":"2024-04-16T15:00:15.466211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.460213+0000","last_change":"2024-04-16T15:00:15.462035+0000","last_active":"2024-04-16T15:00:15.460213+0000","last_peered":"2024-04-16T15:00:15.460213+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.451648+0000","last_became_peered":"2024-04-16T15:00:15.451648+0000","last_unstale":"2024-04-16T15:00:15.460213+0000","last_undegraded":"2024-04-16T15:00:15.460213+0000","last_fullsized":"2024-04-16T15:00:15.460213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.370386+0000","last_change":"2024-04-16T15:00:13.397763+0000","last_active":"2024-04-16T15:00:16.370386+0000","last_peered":"2024-04-16T15:00:16.370386+0000","last_clean":"2024-04-16T15:00:16.370386+0000","last_became_active":"2024-04-16T15:00:13.390622+0000","last_became_peered":"2024-04-16T15:00:13.390622+0000","last_unstale":"2024-04-16T15:00:16.370386+0000","last_undegraded":"2024-04-16T15:00:16.370386+0000","last_fullsized":"2024-04-16T15:00:16.370386+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_clean_scrub_stamp":"2024-04-16T15:00:12.336043+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T15:00:22.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T15:00:23.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:23.050+0000 7f7b95e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:23.058+0000 7f7b95e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:23.242 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:23.242 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-16T15:00:23.254 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-16T15:00:21.323525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89892,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.966401"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.455681+0000","last_change":"2024-04-16T15:00:15.457441+0000","last_active":"2024-04-16T15:00:15.455681+0000","last_peered":"2024-04-16T15:00:15.455681+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.444424+0000","last_became_peered":"2024-04-16T15:00:15.444424+0000","last_unstale":"2024-04-16T15:00:15.455681+0000","last_undegraded":"2024-04-16T15:00:15.455681+0000","last_fullsized":"2024-04-16T15:00:15.455681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.445837+0000","last_change":"2024-04-16T15:00:15.447176+0000","last_active":"2024-04-16T15:00:15.445837+0000","last_peered":"2024-04-16T15:00:15.445837+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.441817+0000","last_became_peered":"2024-04-16T15:00:15.441817+0000","last_unstale":"2024-04-16T15:00:15.445837+0000","last_undegraded":"2024-04-16T15:00:15.445837+0000","last_fullsized":"2024-04-16T15:00:15.445837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.375283+0000","last_change":"2024-04-16T15:00:15.451779+0000","last_active":"2024-04-16T15:00:16.375283+0000","last_peered":"2024-04-16T15:00:16.375283+0000","last_clean":"2024-04-16T15:00:16.375283+0000","last_became_active":"2024-04-16T15:00:15.444660+0000","last_became_peered":"2024-04-16T15:00:15.444660+0000","last_unstale":"2024-04-16T15:00:16.375283+0000","last_undegraded":"2024-04-16T15:00:16.375283+0000","last_fullsized":"2024-04-16T15:00:16.375283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.439719+0000","last_change":"2024-04-16T15:00:15.440602+0000","last_active":"2024-04-16T15:00:15.439719+0000","last_peered":"2024-04-16T15:00:15.439719+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.436185+0000","last_became_peered":"2024-04-16T15:00:15.436185+0000","last_unstale":"2024-04-16T15:00:15.439719+0000","last_undegraded":"2024-04-16T15:00:15.439719+0000","last_fullsized":"2024-04-16T15:00:15.439719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.452396+0000","last_change":"2024-04-16T15:00:15.453913+0000","last_active":"2024-04-16T15:00:15.452396+0000","last_peered":"2024-04-16T15:00:15.452396+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.447182+0000","last_became_peered":"2024-04-16T15:00:15.447182+0000","last_unstale":"2024-04-16T15:00:15.452396+0000","last_undegraded":"2024-04-16T15:00:15.452396+0000","last_fullsized":"2024-04-16T15:00:15.452396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.425720+0000","last_change":"2024-04-16T15:00:15.427043+0000","last_active":"2024-04-16T15:00:15.425720+0000","last_peered":"2024-04-16T15:00:15.425720+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.421570+0000","last_became_peered":"2024-04-16T15:00:15.421570+0000","last_unstale":"2024-04-16T15:00:15.425720+0000","last_undegraded":"2024-04-16T15:00:15.425720+0000","last_fullsized":"2024-04-16T15:00:15.425720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.466211+0000","last_change":"2024-04-16T15:00:15.468018+0000","last_active":"2024-04-16T15:00:15.466211+0000","last_peered":"2024-04-16T15:00:15.466211+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.457433+0000","last_became_peered":"2024-04-16T15:00:15.457433+0000","last_unstale":"2024-04-16T15:00:15.466211+0000","last_undegraded":"2024-04-16T15:00:15.466211+0000","last_fullsized":"2024-04-16T15:00:15.466211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.460213+0000","last_change":"2024-04-16T15:00:15.462035+0000","last_active":"2024-04-16T15:00:15.460213+0000","last_peered":"2024-04-16T15:00:15.460213+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.451648+0000","last_became_peered":"2024-04-16T15:00:15.451648+0000","last_unstale":"2024-04-16T15:00:15.460213+0000","last_undegraded":"2024-04-16T15:00:15.460213+0000","last_fullsized":"2024-04-16T15:00:15.460213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.370386+0000","last_change":"2024-04-16T15:00:13.397763+0000","last_active":"2024-04-16T15:00:16.370386+0000","last_peered":"2024-04-16T15:00:16.370386+0000","last_clean":"2024-04-16T15:00:16.370386+0000","last_became_active":"2024-04-16T15:00:13.390622+0000","last_became_peered":"2024-04-16T15:00:13.390622+0000","last_unstale":"2024-04-16T15:00:16.370386+0000","last_undegraded":"2024-04-16T15:00:16.370386+0000","last_fullsized":"2024-04-16T15:00:16.370386+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_clean_scrub_stamp":"2024-04-16T15:00:12.336043+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T15:00:23.255 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T15:00:23.255 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T15:00:23.255 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T15:00:23.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T15:00:23.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:23.367+0000 7f0777c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:23.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:23.374+0000 7f0777c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:23.605 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:23.605 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T15:00:23.618 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T15:00:23.618 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-16T15:00:23.630 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T15:00:23.630 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-thrash-osd', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '372751789be1725ed06fd077d029295c8cab4c35'}} 2024-04-16T15:00:23.630 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T15:00:23.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-16T15:00:23.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:23.739+0000 7f7e88d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:23.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:23.750+0000 7f7e88d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:23.952 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-16T15:00:23.952 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T15:00:23.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T15:00:24.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:24.062+0000 7f6ede6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:24.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:24.070+0000 7f6ede6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:24.256 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-16T15:00:24.268 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T15:00:23.324014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89892,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.966889"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.455681+0000","last_change":"2024-04-16T15:00:15.457441+0000","last_active":"2024-04-16T15:00:15.455681+0000","last_peered":"2024-04-16T15:00:15.455681+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.444424+0000","last_became_peered":"2024-04-16T15:00:15.444424+0000","last_unstale":"2024-04-16T15:00:15.455681+0000","last_undegraded":"2024-04-16T15:00:15.455681+0000","last_fullsized":"2024-04-16T15:00:15.455681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.445837+0000","last_change":"2024-04-16T15:00:15.447176+0000","last_active":"2024-04-16T15:00:15.445837+0000","last_peered":"2024-04-16T15:00:15.445837+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.441817+0000","last_became_peered":"2024-04-16T15:00:15.441817+0000","last_unstale":"2024-04-16T15:00:15.445837+0000","last_undegraded":"2024-04-16T15:00:15.445837+0000","last_fullsized":"2024-04-16T15:00:15.445837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.375283+0000","last_change":"2024-04-16T15:00:15.451779+0000","last_active":"2024-04-16T15:00:16.375283+0000","last_peered":"2024-04-16T15:00:16.375283+0000","last_clean":"2024-04-16T15:00:16.375283+0000","last_became_active":"2024-04-16T15:00:15.444660+0000","last_became_peered":"2024-04-16T15:00:15.444660+0000","last_unstale":"2024-04-16T15:00:16.375283+0000","last_undegraded":"2024-04-16T15:00:16.375283+0000","last_fullsized":"2024-04-16T15:00:16.375283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.439719+0000","last_change":"2024-04-16T15:00:15.440602+0000","last_active":"2024-04-16T15:00:15.439719+0000","last_peered":"2024-04-16T15:00:15.439719+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.436185+0000","last_became_peered":"2024-04-16T15:00:15.436185+0000","last_unstale":"2024-04-16T15:00:15.439719+0000","last_undegraded":"2024-04-16T15:00:15.439719+0000","last_fullsized":"2024-04-16T15:00:15.439719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.452396+0000","last_change":"2024-04-16T15:00:15.453913+0000","last_active":"2024-04-16T15:00:15.452396+0000","last_peered":"2024-04-16T15:00:15.452396+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.447182+0000","last_became_peered":"2024-04-16T15:00:15.447182+0000","last_unstale":"2024-04-16T15:00:15.452396+0000","last_undegraded":"2024-04-16T15:00:15.452396+0000","last_fullsized":"2024-04-16T15:00:15.452396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.425720+0000","last_change":"2024-04-16T15:00:15.427043+0000","last_active":"2024-04-16T15:00:15.425720+0000","last_peered":"2024-04-16T15:00:15.425720+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.421570+0000","last_became_peered":"2024-04-16T15:00:15.421570+0000","last_unstale":"2024-04-16T15:00:15.425720+0000","last_undegraded":"2024-04-16T15:00:15.425720+0000","last_fullsized":"2024-04-16T15:00:15.425720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.466211+0000","last_change":"2024-04-16T15:00:15.468018+0000","last_active":"2024-04-16T15:00:15.466211+0000","last_peered":"2024-04-16T15:00:15.466211+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.457433+0000","last_became_peered":"2024-04-16T15:00:15.457433+0000","last_unstale":"2024-04-16T15:00:15.466211+0000","last_undegraded":"2024-04-16T15:00:15.466211+0000","last_fullsized":"2024-04-16T15:00:15.466211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.460213+0000","last_change":"2024-04-16T15:00:15.462035+0000","last_active":"2024-04-16T15:00:15.460213+0000","last_peered":"2024-04-16T15:00:15.460213+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.451648+0000","last_became_peered":"2024-04-16T15:00:15.451648+0000","last_unstale":"2024-04-16T15:00:15.460213+0000","last_undegraded":"2024-04-16T15:00:15.460213+0000","last_fullsized":"2024-04-16T15:00:15.460213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.370386+0000","last_change":"2024-04-16T15:00:13.397763+0000","last_active":"2024-04-16T15:00:16.370386+0000","last_peered":"2024-04-16T15:00:16.370386+0000","last_clean":"2024-04-16T15:00:16.370386+0000","last_became_active":"2024-04-16T15:00:13.390622+0000","last_became_peered":"2024-04-16T15:00:13.390622+0000","last_unstale":"2024-04-16T15:00:16.370386+0000","last_undegraded":"2024-04-16T15:00:16.370386+0000","last_fullsized":"2024-04-16T15:00:16.370386+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_clean_scrub_stamp":"2024-04-16T15:00:12.336043+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T15:00:24.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T15:00:24.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:24.379+0000 7f1445006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:24.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:24.387+0000 7f1445006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:00:24.572 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-16T15:00:24.585 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T15:00:23.324014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89892,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.966889"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.455681+0000","last_change":"2024-04-16T15:00:15.457441+0000","last_active":"2024-04-16T15:00:15.455681+0000","last_peered":"2024-04-16T15:00:15.455681+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.444424+0000","last_became_peered":"2024-04-16T15:00:15.444424+0000","last_unstale":"2024-04-16T15:00:15.455681+0000","last_undegraded":"2024-04-16T15:00:15.455681+0000","last_fullsized":"2024-04-16T15:00:15.455681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.445837+0000","last_change":"2024-04-16T15:00:15.447176+0000","last_active":"2024-04-16T15:00:15.445837+0000","last_peered":"2024-04-16T15:00:15.445837+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.441817+0000","last_became_peered":"2024-04-16T15:00:15.441817+0000","last_unstale":"2024-04-16T15:00:15.445837+0000","last_undegraded":"2024-04-16T15:00:15.445837+0000","last_fullsized":"2024-04-16T15:00:15.445837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.375283+0000","last_change":"2024-04-16T15:00:15.451779+0000","last_active":"2024-04-16T15:00:16.375283+0000","last_peered":"2024-04-16T15:00:16.375283+0000","last_clean":"2024-04-16T15:00:16.375283+0000","last_became_active":"2024-04-16T15:00:15.444660+0000","last_became_peered":"2024-04-16T15:00:15.444660+0000","last_unstale":"2024-04-16T15:00:16.375283+0000","last_undegraded":"2024-04-16T15:00:16.375283+0000","last_fullsized":"2024-04-16T15:00:16.375283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.439719+0000","last_change":"2024-04-16T15:00:15.440602+0000","last_active":"2024-04-16T15:00:15.439719+0000","last_peered":"2024-04-16T15:00:15.439719+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.436185+0000","last_became_peered":"2024-04-16T15:00:15.436185+0000","last_unstale":"2024-04-16T15:00:15.439719+0000","last_undegraded":"2024-04-16T15:00:15.439719+0000","last_fullsized":"2024-04-16T15:00:15.439719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.452396+0000","last_change":"2024-04-16T15:00:15.453913+0000","last_active":"2024-04-16T15:00:15.452396+0000","last_peered":"2024-04-16T15:00:15.452396+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.447182+0000","last_became_peered":"2024-04-16T15:00:15.447182+0000","last_unstale":"2024-04-16T15:00:15.452396+0000","last_undegraded":"2024-04-16T15:00:15.452396+0000","last_fullsized":"2024-04-16T15:00:15.452396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.425720+0000","last_change":"2024-04-16T15:00:15.427043+0000","last_active":"2024-04-16T15:00:15.425720+0000","last_peered":"2024-04-16T15:00:15.425720+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.421570+0000","last_became_peered":"2024-04-16T15:00:15.421570+0000","last_unstale":"2024-04-16T15:00:15.425720+0000","last_undegraded":"2024-04-16T15:00:15.425720+0000","last_fullsized":"2024-04-16T15:00:15.425720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.466211+0000","last_change":"2024-04-16T15:00:15.468018+0000","last_active":"2024-04-16T15:00:15.466211+0000","last_peered":"2024-04-16T15:00:15.466211+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.457433+0000","last_became_peered":"2024-04-16T15:00:15.457433+0000","last_unstale":"2024-04-16T15:00:15.466211+0000","last_undegraded":"2024-04-16T15:00:15.466211+0000","last_fullsized":"2024-04-16T15:00:15.466211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T15:00:15.460213+0000","last_change":"2024-04-16T15:00:15.462035+0000","last_active":"2024-04-16T15:00:15.460213+0000","last_peered":"2024-04-16T15:00:15.460213+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"2024-04-16T15:00:15.451648+0000","last_became_peered":"2024-04-16T15:00:15.451648+0000","last_unstale":"2024-04-16T15:00:15.460213+0000","last_undegraded":"2024-04-16T15:00:15.460213+0000","last_fullsized":"2024-04-16T15:00:15.460213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T15:00:16.370386+0000","last_change":"2024-04-16T15:00:13.397763+0000","last_active":"2024-04-16T15:00:16.370386+0000","last_peered":"2024-04-16T15:00:16.370386+0000","last_clean":"2024-04-16T15:00:16.370386+0000","last_became_active":"2024-04-16T15:00:13.390622+0000","last_became_peered":"2024-04-16T15:00:13.390622+0000","last_unstale":"2024-04-16T15:00:16.370386+0000","last_undegraded":"2024-04-16T15:00:16.370386+0000","last_fullsized":"2024-04-16T15:00:16.370386+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_clean_scrub_stamp":"2024-04-16T15:00:12.336043+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T15:00:24.585 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T15:00:24.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-16T15:00:24.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:24.695+0000 7f588a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:24.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:24.703+0000 7f588a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:24.888 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 14 2024-04-16T15:00:24.888 INFO:teuthology.orchestra.run.smithi097.stdout:fsid cb83d412-954b-4b45-966f-51f0bfc0a486 2024-04-16T15:00:24.888 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-04-16T15:00:03.012552+0000 2024-04-16T15:00:24.888 INFO:teuthology.orchestra.run.smithi097.stdout:modified 2024-04-16T15:00:16.358359+0000 2024-04-16T15:00:24.888 INFO:teuthology.orchestra.run.smithi097.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:crush_version 3 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:full_ratio 0.95 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:backfillfull_ratio 0.9 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:nearfull_ratio 0.85 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:require_min_compat_client luminous 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:min_compat_client jewel 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:require_osd_release squid 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:stretch_mode_enabled false 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:allow_crimson=true 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:max_osd 4 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.97:6804/2701876556 v2:172.21.15.97:6805/2701876556 exists,up 125ac5fe-95d0-4d08-8c00-e7aba9a62e13 2024-04-16T15:00:24.889 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.97:6801/2873842391 v2:172.21.15.97:6802/2873842391 exists,up e17fd5ff-0cc5-4287-aca6-1e8916306c65 2024-04-16T15:00:24.890 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.186:6804/4142591846 v2:172.21.15.186:6805/4142591846 exists,up 8b186edc-9c46-4545-9f0f-90a8a3372477 2024-04-16T15:00:24.890 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.186:6808/702210044 v2:172.21.15.186:6809/702210044 exists,up 46950121-5123-4075-a5fe-dd05b8c4a353 2024-04-16T15:00:24.901 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.97:6804/2701876556 v2:172.21.15.97:6805/2701876556 exists,up 125ac5fe-95d0-4d08-8c00-e7aba9a62e13', 'osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.97:6801/2873842391 v2:172.21.15.97:6802/2873842391 exists,up e17fd5ff-0cc5-4287-aca6-1e8916306c65', 'osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.186:6804/4142591846 v2:172.21.15.186:6805/4142591846 exists,up 8b186edc-9c46-4545-9f0f-90a8a3372477', 'osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.186:6808/702210044 v2:172.21.15.186:6809/702210044 exists,up 46950121-5123-4075-a5fe-dd05b8c4a353'] 2024-04-16T15:00:24.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mon.a", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster": "ceph", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "admin_socket_mode": "", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_client_required": "cephx, none", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_cluster_required": "cephx", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_debug": "true", 2024-04-16T15:00:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_service_required": "cephx", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_supported": "cephx", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio": "true", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio_poll_ms": "250", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio_reap_max": "16", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_async_discard_threads": "0", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_block_size": "4096", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_aio": "false", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-16T15:00:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_inflight_ios": "false", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_enable_discard": "false", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_flock_retry": "3", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_inject_crash": "0", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_ioring": "false", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_ioring_hipri": "false", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_type": "", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_alloc_size": "1048576", 2024-04-16T15:00:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_allocator": "hybrid", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_buffered_io": "true", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_check_for_zeros": "false", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_compact_log_sync": "false", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_max_log_runway": "4194304", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_max_prefetch": "1048576", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_min_flush_size": "524288", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_min_log_runway": "1048576", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_replay_recovery": "false", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-16T15:00:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_sync_write": "false", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_allocation_from_file": "true", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_allocator": "hybrid", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_create": "true", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_db_create": "false", 2024-04-16T15:00:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_db_path": "", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_db_size": "0", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_path": "", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_preallocate_file": "false", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_size": "107374182400", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_wal_create": "false", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_wal_path": "", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_wal_size": "100663296", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs": "true", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T15:00:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_autotune": "true", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_size": "0", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_type": "2q", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_clone_cow": "true", 2024-04-16T15:00:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_mode": "none", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_csum_type": "crc32c", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_freelist": "false", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-16T15:00:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_small_allocations": "0", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_default_buffered_read": "true", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_default_buffered_write": "false", 2024-04-16T15:00:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fail_eio": "false", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-16T15:00:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mount": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_umount": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_ignore_data_csum": "false", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-16T15:00:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_log_op_age": "5.000000", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_alloc_size": "0", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_blob_size": "0", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_deferred_txc": "32", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_min_alloc_size": "0", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_nid_prealloc": "1024", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-16T15:00:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_retry_disk_reads": "3", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_cf": "true", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_mem": "512", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-16T15:00:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_tracing": "false", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-16T15:00:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_zero_block_detection": "false", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-16T15:00:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_cluster_require_signatures": "false", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_cluster_require_version": "2", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_require_signatures": "false", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_require_version": "2", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_service_require_signatures": "false", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_service_require_version": "2", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_sign_messages": "true", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "chdir": "", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "client_acl_type": "", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "client_asio_thread_count": "2", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "client_cache_mid": "0.750000", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "client_cache_size": "16384", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "client_caps_release_delay": "5", 2024-04-16T15:00:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "client_check_pool_perm": "true", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_debug_force_sync_read": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_debug_getattr_caps": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_debug_inject_tick_delay": "0", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_die_on_failed_remount": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_dirsize_rbytes": "true", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_force_lazyio": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_fs": "", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_inject_release_failure": "false", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_max_inline_size": "4096", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mds_namespace": "", 2024-04-16T15:00:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "client_metadata": "", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mount_gid": "-1", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mount_timeout": "300", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mount_uid": "-1", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mountpoint": "/", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_notify_timeout": "10", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc": "true", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_max_dirty": "104857600", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_max_objects": "1000", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_size": "209715200", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_target_dirty": "8388608", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_permissions": "true", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_quota": "true", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_quota_df": "true", 2024-04-16T15:00:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_readahead_max_bytes": "0", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_readahead_max_periods": "4", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_readahead_min": "131072", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_reconnect_stale": "false", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_shutdown_timeout": "30", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_snapdir": ".snap", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_tick_interval": "1", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_trace": "", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_try_dentry_invalidate": "false", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_use_faked_inos": "false", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_use_random_mds": "false", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_graylog": "false", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_graylog_port": "12201", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_monitors": "default=true", 2024-04-16T15:00:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_syslog": "false", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_syslog_level": "info", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "-", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_network": "", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_network_interface": "", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zlib_isal": "false", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zlib_level": "5", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zlib_winsize": "-15", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zstd_level": "1", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_alien_op_num_threads": "6", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-16T15:00:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_osd_stat_interval": "0", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_seastar_cpu_cores": "", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_location": "", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_location_hook": "", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_location_hook_timeout": "10", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "daemonize": "false", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "daos_pool": "tank", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_alienstore": "0/5", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_allow_any_pool_priority": "false", 2024-04-16T15:00:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_asok": "1/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_asok_assert_abort": "false", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_asserts_on_shutdown": "true", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_auth": "20/20", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_bdev": "1/3", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_bluefs": "1/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_bluestore": "1/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_buffer": "0/1", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_ceph_exporter": "1/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_cephfs_mirror": "0/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_cephsqlite": "0/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_client": "0/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_compressor": "1/5", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_context": "0/1", 2024-04-16T15:00:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_crimson_interrupt": "0/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_crush": "1/1", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_crypto": "1/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_cyanstore": "0/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_deliberately_leak_memory": "false", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_disable_randomized_ping": "false", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_dpdk": "1/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_eventtrace": "1/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_filer": "0/1", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_filestore": "1/3", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_finisher": "1/1", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_fuse": "1/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_heartbeat_testing_span": "0", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_heartbeatmap": "1/5", 2024-04-16T15:00:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_javaclient": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_journal": "1/3", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_journaler": "0/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_kstore": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_lockdep": "0/1", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mclock": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_balancer": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_locker": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_log": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_log_expire": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_migrator": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_quiesce": "3/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_memstore": "1/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mgr": "2/5", 2024-04-16T15:00:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mgrc": "1/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mon": "20/20", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_monc": "0/10", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_ms": "1/1", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_none": "0/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_objclass": "0/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_objectcacher": "0/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_objecter": "0/1", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_optracker": "0/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_osd": "1/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_paxos": "20/20", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_perfcounter": "1/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_prioritycache": "1/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rados": "0/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd": "0/5", 2024-04-16T15:00:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd_mirror": "0/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd_pwl": "0/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd_replay": "0/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_refs": "0/0", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_reserver": "1/1", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_access": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_datacache": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_dbstore": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_flight": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_notification": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_sync": "1/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rocksdb": "4/5", 2024-04-16T15:00:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_backref": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_cache": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_cleaner": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_device": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_epm": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_journal": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_lba": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_odata": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_omap": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_onode": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_t": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_tm": "0/5", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_striper": "0/1", 2024-04-16T15:00:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_test": "0/5", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_throttle": "1/1", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_timer": "0/1", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_tp": "0/5", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_trace": "1/5", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "device_failure_prediction_mode": "none", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_graylog": "false", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_journald": "false", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_stderr": "true", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_syslog": "false", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event_tracing": "false", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_addr": "0.0.0.0", 2024-04-16T15:00:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_http_port": "9926", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_prio_limit": "5", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_sort_metrics": "true", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_stats_period": "5", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "fake_statfs_for_testing": "0", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "fatal_signal_handlers": "true", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filer_max_purge_ops": "10", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filer_max_truncate_ops": "128", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_apply_finisher_threads": "1", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_blackhole": "false", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_btrfs_clone_range": "true", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_btrfs_snap": "true", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_caller_concurrency": "10", 2024-04-16T15:00:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_collect_device_partition_information": "true", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_commit_timeout": "600.000000", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_inject_read_err": "false", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_omap_check": "false", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_verify_split": "false", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_dump_file": "", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fadvise": "true", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fail_eio": "true", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fd_cache_shards": "16", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fd_cache_size": "128", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fiemap": "false", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fiemap_threshold": "4096", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-16T15:00:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_inject_stall": "0", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_journal_parallel": "false", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_journal_trailing": "false", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_journal_writeahead": "false", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_kill_at": "0", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs": "0", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-16T15:00:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size": "0", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_merge_threshold": "-10", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_odsync_write": "false", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_omap_backend": "rocksdb", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_omap_backend_path": "", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_op_thread_timeout": "60", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_op_threads": "2", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_punch_hole": "false", 2024-04-16T15:00:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_ops": "50", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_seek_data_hole": "false", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_sloppy_crc": "false", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_splice": "false", 2024-04-16T15:00:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_split_multiple": "2", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_split_rand_factor": "20", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_update_to": "1000", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_enable": "true", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-16T15:00:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_xfs_extsize": "false", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_zfs_snap": "false", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fio_dir": "/tmp/fio", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fsid": "cb83d412-954b-4b45-966f-51f0bfc0a486", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_allow_other": "true", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_atomic_o_trunc": "true", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_big_writes": "true", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_debug": "false", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_default_permissions": "false", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_disable_pagecache": "false", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_max_write": "0", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_multithreaded": "true", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_require_active_mds": "true", 2024-04-16T15:00:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_set_user_groups": "true", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_splice_move": "true", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_splice_read": "true", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_splice_write": "true", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_use_invalidate_cb": "true", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "gss_target_name": "ceph", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_file": "", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_inject_failure": "0", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_interval": "5", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "host": "smithi097", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-16T15:00:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "inject_early_sigterm": "false", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "jaeger_agent_port": "6799", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "jaeger_tracing_enable": "false", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_aio": "true", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_align_min_size": "65536", 2024-04-16T15:00:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_block_align": "true", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_block_size": "4096", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_dio": "true", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_discard": "false", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_force_aio": "false", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_ignore_corruption": "false", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_max_write_bytes": "10485760", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_max_write_entries": "100", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_replay_from": "0", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_write_header_frequency": "0", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_zero_on_create": "false", 2024-04-16T15:00:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "journaler_prefetch_periods": "10", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "journaler_prezero_periods": "5", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "journaler_write_head_interval": "15", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "keyfile": "", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_backend": "rocksdb", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_default_stripe_size": "65536", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_fsck_on_mount": "false", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_max_bytes": "67108864", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_max_ops": "512", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_nid_prealloc": "1024", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_onode_map_size": "1024", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_sync_submit_transaction": "false", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_sync_transaction": "false", 2024-04-16T15:00:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "librados_thread_count": "2", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "lockdep": "false", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "lockdep_force_backtrace": "false", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_coarse_timestamps": "true", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_flush_on_exit": "false", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_graylog_host": "127.0.0.1", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_graylog_port": "12201", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_max_new": "1000", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_max_recent": "10000", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_stderr_prefix": "", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_stop_at_utilization": "0.970000", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": "true", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_graylog": "false", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_journald": "false", 2024-04-16T15:00:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_stderr": "false", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_syslog": "false", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "max_rotating_auth_attempts": "10", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_action_on_write_error": "1", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_alternate_name_max": "8192", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_asio_thread_count": "2", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_export_pin": "true", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_dirs": "true", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_interval": "5", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_interval": "10", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_max": "-1", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_max_until": "-1", 2024-04-16T15:00:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_merge_size": "50", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_midchunk": "0.300000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_min_start": "0.200000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_minchunk": "0.001000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_mode": "0", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_need_max": "1.200000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_need_min": "0.800000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_overload_epochs": "2", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_bits": "3", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_size": "10000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-16T15:00:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_target_decay": "10.000000", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_beacon_grace": "15.000000", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_beacon_interval": "4.000000", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_mid": "0.700000", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_delay": "0", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_release_free_interval": "10", 2024-04-16T15:00:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_reservation": "0.050000", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_trim_interval": "1", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_trim_threshold": "262144", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_client_prealloc_inos": "1000", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_connect_bootstrapping": "false", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_damage_table_max_entries": "10000", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_auth_pins": "false", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_frag": "false", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_scatterstat": "false", 2024-04-16T15:00:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_subtrees": "false", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_decay_halflife": "5.000000", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_default_dir_hash": "2", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_defer_session_stale": "true", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_deny_all_reconnect": "false", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_keys_per_op": "16384", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_max_commit_size": "10", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_max_entries": "0", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_prefetch": "true", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_on_map": "false", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_early_reply": "true", 2024-04-16T15:00:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_enable_op_tracker": "true", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_enforce_unique_name": "true", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_random": "true", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_health_summarize_threshold": "10", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-16T15:00:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_health_dummy": "false", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_migrator_session_race": "false", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_join_fs": "", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_journal_format": "1", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_create_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_export_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_import_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_journal_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_journal_expire_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_journal_replay_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_link_at": "0", 2024-04-16T15:00:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_mdstable_at": "0", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_openc_at": "0", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_rename_at": "0", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_shutdown_at": "0", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_event_large_threshold": "524288", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_events_per_segment": "1024", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_max_events": "-1", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_max_segments": "128", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_pause": "false", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_segment_size": "0", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-16T15:00:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_trim_threshold": "128", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_warn_factor": "2.000000", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_caps_per_client": "1048576", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_completed_flushes": "100000", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_completed_requests": "100000", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_export_size": "20971520", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_file_recover": "32", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_purge_files": "64", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_purge_ops": "8192", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_snaps_per_dir": "100", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_metrics_update_interval": "2", 2024-04-16T15:00:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_min_caps_per_client": "100", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_min_caps_working_set": "10000", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_numa_node": "-1", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_complaint_time": "30.000000", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_duration": "600", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_size": "20", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_slow_op_size": "20", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_log_threshold": "5", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_ping_grace": "15", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_ping_interval": "5", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-16T15:00:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_max_caps": "30000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_warning_threshold": "262144", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_replay_interval": "1.000000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_root_ino_gid": "0", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_root_ino_uid": "0", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-16T15:00:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_shutdown_check": "0", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_skip_ino": "0", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_snap_max_uid": "4294967294", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_snap_min_uid": "0", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_snap_rstat": "false", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_standby_replay_damaged": "false", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_symlink_recovery": "true", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-16T15:00:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_thrash_exports": "0", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_thrash_fragments": "0", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_tick_interval": "5.000000", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_valgrind_exit": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_verify_backtrace": "1", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_verify_scatter": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_wipe_sessions": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool_debug": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_device_bytes": "1073741824", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_page_set": "false", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_page_size": "65536", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_client_bytes": "134217728", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_client_messages": "512", 2024-04-16T15:00:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_disabled_modules": "", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_max_pg_creating": "1024", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_max_pg_num_change": "128", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mds_bytes": "134217728", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mds_messages": "128", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mon_bytes": "134217728", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mon_messages": "128", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_osd_bytes": "536870912", 2024-04-16T15:00:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_osd_messages": "8192", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_pool": "true", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_standby_modules": "true", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_stats_period": "5", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_stats_threshold": "5", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_tick_period": "2", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_allow_pool_delete": "true", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_allow_pool_size_one": "true", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_auth_validate_all_caps": "true", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_bytes": "104857600", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_directed_command_retry": "2", 2024-04-16T15:00:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_parallel": "3", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_log_interval": "1.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_ping_interval": "10.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_target_rank": "-1", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_file_level": "debug", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_file": "true", 2024-04-16T15:00:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_journald": "false", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_compact_on_bootstrap": "false", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_compact_on_start": "false", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_compact_on_trim": "true", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-16T15:00:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cpu_threads": "4", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_crush_min_required_version": "hammer", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_daemon_bytes": "419430400", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data_avail_crit": "5", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data_avail_warn": "5", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data_size_warn": "16106127360", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_dump_json": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_dump_transactions": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_extra_checks": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-16T15:00:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_require_reef": "false", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_require_squid": "false", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_down_mkfs_grace": "120", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_down_uptime_grace": "60", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_election_default_strategy": "1", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_election_timeout": "5.000000", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_elector_ping_divisor": "2", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_enable_op_tracker": "true", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_fake_pool_delete": "false", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_force_quorum_join": "false", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_globalid_prealloc": "10000", 2024-04-16T15:00:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_detail_to_clog": "false", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_log_update_period": "5", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_max_detail": "50", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_to_clog": "true", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_to_clog_interval": "600", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_host": "172.21.15.97", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_host_override": "", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_initial_members": "", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_lease": "5.000000", 2024-04-16T15:00:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_log_full_interval": "50", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_log_max": "10000", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_log_max_summary": "50", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_log_epochs": "500", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_osd": "10000", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_pg_per_osd": "10000", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_pool_pg_num": "65536", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mds_force_trim_to": "0", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mds_skip_sanity": "false", 2024-04-16T15:00:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_memory_autotune": "true", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_memory_target": "2147483648", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_beacon_grace": "30", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_digest_period": "5", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_inactive_grace": "60", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_min_osdmap_epochs": "50", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_complaint_time": "30", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_duration": "600", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_size": "20", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_slow_op_size": "20", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_log_threshold": "5", 2024-04-16T15:00:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_auto_mark_in": "false", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_cache_size": "500", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_down_out_interval": "600", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_force_trim_to": "0", 2024-04-16T15:00:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_min_down_reporters": "2", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-16T15:00:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_report_timeout": "900", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_stuck_threshold": "60", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-16T15:00:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_probe_timeout": "2.000000", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_max_change": "0.050000", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_max_osds": "4", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_interval": "86400", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_max_keys": "100", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_timeout": "300", 2024-04-16T15:00:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_session_timeout": "300", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_smart_report_timeout": "5", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stat_smooth_intervals": "6", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_pool_min_size": "2", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_pool_size": "4", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_debug": "false", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_provider_kill_at": "0", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_requester_kill_at": "0", 2024-04-16T15:00:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_timeout": "60.000000", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_target_pg_per_osd": "100", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_tick_interval": "5", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_timecheck_interval": "300.000000", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_use_min_delay_socket": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_older_version_delay": "604800", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_misplaced": "false", 2024-04-16T15:00:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_older_version": "true", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "monmap": "", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-16T15:00:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_tracing_enabled": "false", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_op_threads": "3", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_cm": "false", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_device_name": "", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_dscp": "96", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_local_gid": "", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_port_num": "1", 2024-04-16T15:00:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_sl": "3", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_support_srq": "true", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_type": "ib", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_reap_threshold": "5", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_before_connect": "false", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_ipv4": "true", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_ipv6": "false", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_msgr1": "true", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_msgr2": "true", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_port_max": "7568", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_port_min": "6800", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-16T15:00:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_retry_count": "3", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_retry_delay": "5", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_client": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_mds": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_mgr": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_mon": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_osd": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_client_mode": "crc", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_cluster_mode": "crc", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_cluster_type": "", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_compress_secure": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_connection_idle_timeout": "900", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_connection_ready_timeout": "10", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_crc_data": "true", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_crc_header": "true", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_bad_msg": "false", 2024-04-16T15:00:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_bug": "true", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_old_message": "true", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_skipped_message": "false", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_coremask": "0xF", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_enable_tso": "true", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_hugepages": "", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_lro": "true", 2024-04-16T15:00:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_memory_channel": "4", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_pmd": "", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_port_id": "0", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dump_on_send": "false", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_initial_backoff": "0.200000", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_delay_max": "1.000000", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_delay_type": "", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_network_congestion": "0", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_socket_failures": "0", 2024-04-16T15:00:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_learn_addr_from_peer": "true", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_max_accept_failures": "4", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_max_backoff": "15.000000", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_mon_client_mode": "crc", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_mon_cluster_mode": "crc", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_mon_service_mode": "crc", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_osd_compress_min_size": "1024", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_osd_compress_mode": "none", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_pq_min_cost": "65536", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_public_type": "", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_service_mode": "crc", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_listen_backlog": "512", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_nodelay": "true", 2024-04-16T15:00:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_rcvbuf": "0", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_type": "async+posix", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "no_config_file": "false", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_completion_locks_per_session": "32", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_inflight_ops": "1024", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_tick_interval": "5.000000", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_timeout": "10.000000", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objectstore_blackhole": "false", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "openssl_engine_opts": "", 2024-04-16T15:00:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_delay_time": "5.000000", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_hist_halflife": "1000", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_max_low_ops": "2", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_max_ops": "4", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_slop": "0.020000", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_asio_thread_count": "2", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_async_recovery_min_cost": "100", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backfill_scan_max": "512", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backfill_scan_min": "64", 2024-04-16T15:00:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backoff_on_degraded": "false", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backoff_on_peering": "false", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backoff_on_unfound": "true", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_beacon_report_interval": "300", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_duration": "30", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_max_block_size": "67108864", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_blkin_trace_all": "false", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-16T15:00:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_check_for_log_corruption": "false", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_update_on_start": "true", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_message_cap": "256", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_message_size_cap": "524288000", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_op_priority": "63", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_watch_timeout": "30", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_command_max_records": "256", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_compact_on_start": "false", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-16T15:00:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_update_on_start": "true", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_update_weight_set": "true", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_feed_pullee": "-1", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_misdirected_ops": "false", 2024-04-16T15:00:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_no_acting_change": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_no_purge_strays": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_op_order": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_shutdown": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_trim_objects": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_snaps": "false", 2024-04-16T15:00:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_keys": "1024", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_stride": "524288", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_default_notify_timeout": "30", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep": "0.000000", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-16T15:00:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_discard_disconnected_ops": "true", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_enable_op_tracker": "true", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_info": "true", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_shutdown": "true", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-16T15:00:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_function_tracing": "false", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_grace": "20", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_interval": "6", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_min_peers": "10", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_min_size": "2000", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_stale": "600", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_hit_set_max_size": "100000", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_hit_set_min_size": "1000", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-16T15:00:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_journal_size": "5120", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_kill_backfill_at": "0", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_cache_size": "50", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_dedup": "true", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_message_max": "40", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_share_max_epochs": "40", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_attr_name_len": "100", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_attr_size": "0", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_backfills": "1", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_markdown_count": "5", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_markdown_period": "600", 2024-04-16T15:00:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_object_name_len": "2048", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_object_namespace_len": "256", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_object_size": "134217728", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pg_blocked_by": "16", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pg_log_entries": "10000", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pgls": "1024", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_push_cost": "8388608", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_push_objects": "10", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_scrubs": "3", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_trimming_pgs": "2", 2024-04-16T15:00:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_write_op_reply_len": "64", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_write_size": "90", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_profile": "balanced", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-16T15:00:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_base": "805306368", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_cache_min": "134217728", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_target": "4294967296", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_target_autotune": "false", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_min_pg_log_entries": "250", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_min_recovery_priority": "0", 2024-04-16T15:00:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_report_interval": "5", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_num_cache_shards": "32", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_num_op_tracker_shard": "32", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_numa_auto_affinity": "true", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_numa_node": "-1", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_numa_prefer_iface": "true", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objecter_finishers": "1", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objectstore": "bluestore", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objectstore_fuse": "false", 2024-04-16T15:00:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objectstore_tracing": "false", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_complaint_time": "30.000000", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_duration": "600", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_size": "20", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_slow_op_size": "20", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_log_threshold": "5", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_shards": "0", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_shards_hdd": "5", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_shards_ssd": "8", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-16T15:00:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_pq_min_cost": "65536", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_queue_cut_off": "high", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_thread_timeout": "15", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_open_classes_on_start": "true", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_os_flags": "0", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_peering_op_priority": "255", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_delete_cost": "1048576", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_delete_priority": "5", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_log_trim_max": "10000", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_log_trim_min": "100", 2024-04-16T15:00:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_crimson": "true", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-16T15:00:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flags": "0", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_min_size": "2", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_pg_num": "32", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_pgp_num": "0", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_read_ratio": "70", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_size": "3", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_type": "replicated", 2024-04-16T15:00:25.105 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_push_per_object_cost": "1000", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recover_clone_overlap": "true", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_cost": "20971520", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_active": "0", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_single_start": "1", 2024-04-16T15:00:25.106 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_op_priority": "3", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_priority": "5", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep": "0.000000", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_repair_during_recovery": "false", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_requested_scrub_priority": "120", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_auto_repair": "false", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-16T15:00:25.107 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_begin_hour": "0", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_begin_week_day": "0", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_chunk_max": "25", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_chunk_min": "5", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_cost": "52428800", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_during_recovery": "false", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_end_hour": "0", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_end_week_day": "0", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_event_cost": "4096", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_invalid_stats": "true", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-16T15:00:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_max_preemptions": "5", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_priority": "5", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_sleep": "0.000000", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_skip_data_digest": "false", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_smart_report_timeout": "5", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_cost": "1048576", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_priority": "5", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-16T15:00:25.109 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_target_transaction_size": "30", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-16T15:00:25.110 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tracing": "false", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_use_stale_snap": "false", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "osdc_blkin_trace_all": "false", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_kill_at": "0", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_max_join_drift": "10", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_min": "500", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_min_wait": "0.050000", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_propose_interval": "1.000000", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_service_trim_max": "500", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_service_trim_min": "10", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_stash_full_interval": "25", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_trim_max": "500", 2024-04-16T15:00:25.111 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_trim_min": "250", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "perf": "true", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "-", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrv": "", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "public_bind_addr": "-", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "public_network": "", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "public_network_interface": "", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "qat_compressor_enabled": "false", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "qat_compressor_session_max_number": "256", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "rados_mon_op_timeout": "0", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "rados_osd_op_timeout": "0", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "rados_tracing": "false", 2024-04-16T15:00:25.112 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_atime_update_interval": "60", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_balance_parent_reads": "false", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_balance_snap_reads": "false", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_blkin_trace_all": "false", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache": "true", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_policy": "writearound", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_size": "33554432", 2024-04-16T15:00:25.113 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_clone_copy_on_read": "false", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_compression_hint": "none", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_concurrent_management_ops": "10", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_clone_format": "auto", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_data_pool": "", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_features": "61", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_format": "2", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_map_options": "", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_order": "22", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_pool": "rbd", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_stripe_count": "0", 2024-04-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_stripe_unit": "0", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_enable_alloc_hint": "true", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_io_scheduler": "simple", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_order": "24", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_pool": "", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_splay_width": "4", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_localize_parent_reads": "false", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_localize_snap_reads": "false", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-16T15:00:25.117 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mtime_update_interval": "60", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_non_blocking_aio": "true", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_op_thread_timeout": "60", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_op_threads": "1", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_parent_cache_enabled": "false", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-16T15:00:25.118 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_plugins": "", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_bps_burst": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_bps_limit": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_exclude_ops": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_iops_burst": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_iops_limit": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-16T15:00:25.119 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_read_from_replica_policy": "default", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_skip_partial_discard": "true", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_tracing": "false", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_validate_names": "true", 2024-04-16T15:00:25.120 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_validate_pool": "true", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "restapi_base_url": "", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "restapi_log_level": "", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_acl_grants_max_num": "100", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_admin_entry": "admin", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_backend_store": "rados", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_barbican_url": "", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_beast_enable_async": "true", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_counters_cache": "false", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-16T15:00:25.121 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cache_enabled": "true", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cache_expiry_interval": "900", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cache_lru_size": "10000", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_config_store": "rados", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_content_length_compat": "false", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_copy_obj_progress": "true", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cors_rules_max_num": "100", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cross_domain_policy": "", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_addr": "", 2024-04-16T15:00:25.122 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_password": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_username": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_require_ssl": "true", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-16T15:00:25.123 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_addr": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_auth": "token", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_namespace": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_prefix": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-16T15:00:25.124 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_token_file": "", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_buffersize": "524288", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_low_speed_time": "300", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-16T15:00:25.125 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_port": "6379", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_changes_size": "1000", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_num_shards": "128", 2024-04-16T15:00:25.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_window": "30", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-16T15:00:25.127 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dns_name": "", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dns_s3website_name": "", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dynamic_resharding": "true", 2024-04-16T15:00:25.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_gc_threads": "true", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_lc_threads": "true", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_ops_log": "false", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_quota_threads": "true", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_static_website": "false", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_usage_log": "false", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enforce_swift_acls": "true", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_exit_timeout_secs": "120", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_expose_bucket": "false", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_extended_http_attrs": "", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_filter": "none", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_frontends": "beast port=7480", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_deferred": "50", 2024-04-16T15:00:25.129 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_objs": "32", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_processor_period": "3600", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_init_timeout": "300", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-16T15:00:25.130 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_kafka_connection_idle": "300", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_domain": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_password": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_password_path": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_project": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_tenant": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_token": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_token_path": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_user": "", 2024-04-16T15:00:25.131 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_api_version": "2", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_domain": "", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_password": "", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_project": "", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_user": "", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_url": "", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_debug_interval": "-1", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_lock_max_time": "90", 2024-04-16T15:00:25.132 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_objs": "32", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_rules": "1000", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_worker": "3", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_thread_delay": "0", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_dnattr": "uid", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_searchfilter": "", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_uri": "", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-16T15:00:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_http_headers": "", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_object_name_utc": "false", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_attr_name_len": "0", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_attr_size": "0", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_chunk_size": "4194304", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_listing_results": "1000", 2024-04-16T15:00:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_notify_retries": "10", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_put_param_size": "1048576", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_put_size": "5368709120", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_slo_entries": "1000", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_md_log_max_shards": "64", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_mp_lock_max_time": "600", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-16T15:00:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_max_gc": "300", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_num_async_rados_threads": "32", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_num_control_oids": "8", 2024-04-16T15:00:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_numa_node": "-1", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_objexp_chunk_size": "100", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_objexp_gc_interval": "600", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_op_thread_timeout": "600", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_op_tracing": "false", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_opa_token": "", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_opa_url": "", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_opa_verify_ssl": "true", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-16T15:00:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_rados": "false", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_socket_path": "", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_push_interval": "2.000000", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_lanes": "3", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_partitions": "3", 2024-04-16T15:00:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_print_continue": "true", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rados_tracing": "false", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_realm": "", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_realm_id": "", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_region": "", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-16T15:00:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_request_uri": "", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_batch_size": "64", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_max_aio": "128", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_num_logs": "16", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_thread_interval": "600", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_resolve_cname": "false", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_run_sync_thread": "true", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-16T15:00:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_scheduler_type": "throttler", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_script_uri": "", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_service_provider_name": "", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_client_id": "", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_client_secret": "", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_entry": "sts", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_key": "sts", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_min_session_duration": "900", 2024-04-16T15:00:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_token_introspection_url": "", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_account_in_url": "false", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_auth_entry": "auth", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_auth_url": "", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_custom_header": "", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_need_stats": "true", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_tenant_name": "", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_token_expiration": "86400", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_url": "", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_url_prefix": "swift", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-16T15:00:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_lease_period": "120", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_thread_pool_size": "512", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-16T15:00:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_comment": "", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_createby": "", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_encoding": "", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_flag": "false", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_origin": "", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_tracker": "", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_trust_forwarded_https": "false", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_max_shards": "32", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_max_user_shards": "1", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_use_opa_authz": "false", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_counters_cache": "false", 2024-04-16T15:00:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_max_buckets": "1000", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_policies_max_num": "100", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_unique_email": "true", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_verify_ssl": "true", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zone": "", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zone_id": "", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-16T15:00:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zonegroup": "", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zonegroup_id": "", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rgwlc_auto_session_clear": "true", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_block_size": "4096", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_size": "536870912", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-16T15:00:25.146 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_index_type": "binary_search", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_partition_filters": "false", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_perf": "false", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "run_dir": "/var/run/ceph", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_block_create": "true", 2024-04-16T15:00:25.147 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_cache_lru_size": "67108864", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_default_max_object_size": "16777216", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_device_size": "53687091200", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_batch_capacity": "16", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_main_device_type": "SSD", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-16T15:00:25.148 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_segment_size": "67108864", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "set_keepcaps": "false", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "setgroup": "", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "setuser": "", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "setuser_match_path": "", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "thp": "false", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "threadpool_default_timeout": "60", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout: "throttler_perf_counter": "true" 2024-04-16T15:00:25.149 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-16T15:00:25.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-16T15:00:25.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:25.262+0000 7f4143a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:25.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:25.268+0000 7f4143a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:25.280 INFO:teuthology.orchestra.run.smithi097.stdout:{} 2024-04-16T15:00:25.280 INFO:teuthology.orchestra.run.smithi097.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-16T15:00:25.291 DEBUG:teuthology.orchestra.run.smithi097:> type ceph-objectstore-tool 2024-04-16T15:00:25.306 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T15:00:25.306 DEBUG:teuthology.orchestra.run.smithi186:> type ceph-objectstore-tool 2024-04-16T15:00:25.324 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T15:00:25.325 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-16T15:00:25.335 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-16T15:00:25.336 INFO:tasks.radosbench:Beginning radosbench... 2024-04-16T15:00:25.336 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-16T15:00:25.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-16T15:00:25.337 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-16T15:00:25.337 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T15:00:25.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:00:25.337 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3] 2024-04-16T15:00:25.337 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T15:00:25.337 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:00:25.337 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-16T15:00:25.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:00:25.414 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T15:00:25.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:25.460+0000 7f45c0a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:25.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:25.465+0000 7f45c0a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:25.460+0000 7fc6622ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:25.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:25.472+0000 7fc6622ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'unique_pool_0' created 2024-04-16T15:00:26.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-16T15:00:26.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:26.542+0000 7f311682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:26.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:26.556+0000 7f311682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-16T15:00:28.421 DEBUG:teuthology.orchestra.run.smithi097:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-16T15:00:28.424 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-16T15:00:28.424 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-16T15:00:28.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:hints = 1 2024-04-16T15:00:28.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-16T15:00:28.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:Object prefix: benchmark_data_smithi097_31987 2024-04-16T15:00:28.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:00:28.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 0 0 0 0 0 0 - 0 2024-04-16T15:00:29.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1 16 19 3 0.187479 0.1875 0.0379486 0.0336668 2024-04-16T15:00:30.414 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-16T15:00:30.415 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:00:30.415 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T15:00:30.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T15:00:30.489 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2 16 19 3 0.0937395 0 - 0.0336668 2024-04-16T15:00:30.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.534+0000 7f31be82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.543+0000 7f31be82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:no valid command found; 10 closest matches: 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:config get 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:config help 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:config set ... 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:config show 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:deep_scrub 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:dump_historic_ops 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:dump_historic_slow_ops 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:dump_metrics [] 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:dump_ops_in_flight 2024-04-16T15:00:30.559 INFO:teuthology.orchestra.run.smithi097.stderr:dump_pgstate_history 2024-04-16T15:00:30.560 INFO:teuthology.orchestra.run.smithi097.stderr:Error EINVAL: invalid command 2024-04-16T15:00:30.560 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:00:30.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T15:00:30.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.670+0000 7f5ecc0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.679+0000 7f5ecc0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.691 INFO:teuthology.orchestra.run.smithi097.stderr:no valid command found; 10 closest matches: 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:config get 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:config help 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:config set ... 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:config show 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:deep_scrub 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:dump_historic_ops 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:dump_historic_slow_ops 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:dump_metrics [] 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:dump_ops_in_flight 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:dump_pgstate_history 2024-04-16T15:00:30.692 INFO:teuthology.orchestra.run.smithi097.stderr:Error EINVAL: invalid command 2024-04-16T15:00:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:00:30.695 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:00:30.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T15:00:30.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.805+0000 7f2781bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.814+0000 7f2781bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:no valid command found; 10 closest matches: 2024-04-16T15:00:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:config get 2024-04-16T15:00:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:config help 2024-04-16T15:00:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:config set ... 2024-04-16T15:00:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:config show 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:deep_scrub 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:dump_historic_ops 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:dump_historic_slow_ops 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:dump_metrics [] 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:dump_ops_in_flight 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:dump_pgstate_history 2024-04-16T15:00:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:Error EINVAL: invalid command 2024-04-16T15:00:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:00:30.831 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:00:30.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T15:00:30.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.941+0000 7f42508ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:30.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:30.949+0000 7f42508ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:31.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3 16 19 3 0.0624921 0 - 0.0336668 2024-04-16T15:00:32.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 4 16 19 3 0.0468689 0 - 0.0336668 2024-04-16T15:00:33.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 5 16 19 3 0.0374948 0 - 0.0336668 2024-04-16T15:00:34.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 6 16 19 3 0.0312456 0 - 0.0336668 2024-04-16T15:00:35.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 7 16 19 3 0.0267821 0 - 0.0336668 2024-04-16T15:00:36.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 8 16 19 3 0.0234343 0 - 0.0336668 2024-04-16T15:00:37.490 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 9 16 19 3 0.0208304 0 - 0.0336668 2024-04-16T15:00:38.491 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 10 16 19 3 0.0187474 0 - 0.0336668 2024-04-16T15:00:39.491 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 11 16 19 3 0.017043 0 - 0.0336668 2024-04-16T15:00:40.491 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 12 16 19 3 0.0156228 0 - 0.0336668 2024-04-16T15:00:41.491 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 13 16 19 3 0.014421 0 - 0.0336668 2024-04-16T15:00:42.491 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 14 16 19 3 0.0133909 0 - 0.0336668 2024-04-16T15:00:43.491 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 15 16 19 3 0.0124982 0 - 0.0336668 2024-04-16T15:00:44.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 16 16 19 3 0.011717 0 - 0.0336668 2024-04-16T15:00:45.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 17 16 19 3 0.0110278 0 - 0.0336668 2024-04-16T15:00:46.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 18 16 19 3 0.0104152 0 - 0.0336668 2024-04-16T15:00:47.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 19 16 19 3 0.00986699 0 - 0.0336668 2024-04-16T15:00:48.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:00:48.494954+0000 min lat: 0.0301502 max lat: 0.0379486 avg lat: 0.0336668 2024-04-16T15:00:48.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:00:48.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 20 16 19 3 0.00937365 0 - 0.0336668 2024-04-16T15:00:49.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 21 16 19 3 0.00892729 0 - 0.0336668 2024-04-16T15:00:50.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 22 16 19 3 0.00852151 0 - 0.0336668 2024-04-16T15:00:51.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 23 16 19 3 0.00815102 0 - 0.0336668 2024-04-16T15:00:52.020 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T15:00:52.020 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T15:00:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:00:52.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:00:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:00:52.026 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:00:52.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 24 16 19 3 0.00781139 0 - 0.0336668 2024-04-16T15:00:53.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 25 16 74 58 0.14498 0.143229 0.11042 6.88882 2024-04-16T15:00:54.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:00:54.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:54.189+0000 7fbd92f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:54.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:54.199+0000 7fbd92f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T15:00:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:00:54.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:00:54.492 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 26 15 305 290 0.69702 14.5 0.0509527 1.43262 2024-04-16T15:00:55.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 27 16 541 525 1.21511 14.6875 0.091895 0.821576 2024-04-16T15:00:56.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:00:56.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 28 16 775 759 1.69397 14.625 0.0940866 0.589415 2024-04-16T15:00:56.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:56.649+0000 7f52c379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:56.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:56.659+0000 7f52c379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:57.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-16T15:00:57.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:00:57.028 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T15:00:57.028 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T15:00:57.028 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T15:00:57.030 INFO:tasks.ceph.osd.3:Started 2024-04-16T15:00:57.031 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T15:00:57.033 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T15:00:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:00:57.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:00:57.180 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:00:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:00:57.185 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T15:00:57.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 29 15 1014 999 2.15274 15 0.0617839 0.463812 2024-04-16T15:00:58.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 30 16 1183 1167 2.43094 10.5 0.0673179 0.410476 2024-04-16T15:00:59.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:00:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:59.217+0000 7f9a26f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:59.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:00:59.226+0000 7f9a26f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:00:59.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 31 16 1413 1397 2.81617 14.375 0.117611 0.354674 2024-04-16T15:00:59.548 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T15:00:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:00:59.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:01:00.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 32 16 1648 1632 3.1871 14.6875 0.0887268 0.313445 2024-04-16T15:01:01.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 33 16 1823 1807 3.42192 10.9375 0.128007 0.2893 2024-04-16T15:01:01.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:01:01.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:01.665+0000 7f73df864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:01.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:01.672+0000 7f73df864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:02.186 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: "ops_in_flight": [ 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "background_recovery_sub", 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 114, 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": {} 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.355 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 19, 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.218994+0000" 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.219002+0000" 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.219883+0000" 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.220077+0000" 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.223077+0000" 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.223196+0000" 2024-04-16T15:01:02.357 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.127779+0000" 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 26, 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.238678+0000" 2024-04-16T15:01:02.358 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.238688+0000" 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.241771+0000" 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.359 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.241993+0000" 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.243105+0000" 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.127868+0000" 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.360 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 29, 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.250530+0000" 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.250546+0000" 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.250972+0000" 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.361 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.252111+0000" 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.254023+0000" 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.254068+0000" 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.133198+0000" 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 52, 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.340056+0000" 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.363 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.340072+0000" 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.340386+0000" 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.340884+0000" 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.364 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.341294+0000" 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.133297+0000" 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:01:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 88, 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.4", 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 22, 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 21, 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 22 epoch_requested: 21 MLogRec from 2 log log((0'0,20'99], crt=20'100) pi ([15,20] all_participants={0, 2, 3} intervals=[([18,20] acting={0, 2})]) pg_lease(ru 0.000000000s ub 70.809883118s int 16.000000000s) +create_info", 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.253874+0000" 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.366 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.253884+0000" 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.254937+0000" 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.255706+0000" 2024-04-16T15:01:02.367 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.256588+0000" 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.256664+0000" 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.269716+0000" 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.368 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.269785+0000" 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.369 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 100, 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "1", 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.1", 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 22, 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 22, 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 22 epoch_requested: 22 MInfoRec from 1 info: 2.1( empty local-lis/les=21/22 n=0 ec=12/12 lis/c=21/12 les/c/f=22/13/0 sis=21) pg_lease(ru 70.934211731s ub 71.036506653s int 16.000000000s)", 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.370 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.280135+0000" 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.280143+0000" 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.280918+0000" 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.371 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.282587+0000" 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.286399+0000" 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.287466+0000" 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.372 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.287616+0000" 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.373 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 101, 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "1", 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.1", 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 22, 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 22, 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 22 epoch_requested: 22 MInfoRec from 1 info: 2.1( empty local-lis/les=21/22 n=0 ec=12/12 lis/c=21/21 les/c/f=22/22/0 sis=21) pg_lease(ru 70.934211731s ub 71.036506653s int 16.000000000s)", 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.286925+0000" 2024-04-16T15:01:02.374 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.286932+0000" 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.289105+0000" 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.289436+0000" 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.336386+0000" 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.337077+0000" 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.376 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.337325+0000" 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.377 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 109, 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "0", 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "1.0", 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 22, 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 22, 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 22 epoch_requested: 22 MLeaseAck epoch 22 from osd.0 pg_lease_ack(ruub 19.299064636s)", 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.311645+0000" 2024-04-16T15:01:02.378 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.311652+0000" 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.311883+0000" 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.379 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.312303+0000" 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.336495+0000" 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.338821+0000" 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.380 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.339113+0000" 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.381 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 113, 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "1.0", 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 22, 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 22, 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 22 epoch_requested: 22 MLeaseAck epoch 22 from osd.2 pg_lease_ack(ruub 19.299064636s)", 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.320568+0000" 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.382 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.320575+0000" 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.320743+0000" 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.383 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.320862+0000" 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.336626+0000" 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.339069+0000" 2024-04-16T15:01:02.384 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.340258+0000" 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.385 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777238, 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "3", 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.4", 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 22, 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 22, 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 22 epoch_requested: 22 ActivateCommitted from 22 processed at 22", 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.386 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.346073+0000" 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.346138+0000" 2024-04-16T15:01:02.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 17, 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.215697+0000" 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.215726+0000" 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.216353+0000" 2024-04-16T15:01:02.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.217413+0000" 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.220202+0000" 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.390 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.220328+0000" 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.144996+0000" 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 22, 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.391 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.221855+0000" 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.221873+0000" 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.222244+0000" 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.392 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.222386+0000" 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.225046+0000" 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.145150+0000" 2024-04-16T15:01:02.393 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 24, 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.227352+0000" 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.227366+0000" 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.394 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.229011+0000" 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.230573+0000" 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:01.230908+0000" 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:01:02.142361+0000" 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:01:02.396 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-16T15:01:02.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 34 16 1823 1807 3.32128 0 - 0.2893 2024-04-16T15:01:03.493 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 35 16 1823 1807 3.22638 0 - 0.2893 2024-04-16T15:01:03.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:01:03.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:03.894+0000 7fd33719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:03.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:03.900+0000 7fd33719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:04.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 36 16 1832 1816 3.15239 0.1875 2.50983 0.296038 2024-04-16T15:01:05.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 37 16 1832 1816 3.06719 0 - 0.296038 2024-04-16T15:01:06.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 38 16 1832 1816 2.98648 0 - 0.296038 2024-04-16T15:01:07.398 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T15:01:07.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:01:07.398 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2024-04-16T15:01:07.398 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T15:01:07.398 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:01:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:01:07.399 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-04-16T15:01:07.400 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T15:01:07.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 39 16 1832 1816 2.9099 0 - 0.296038 2024-04-16T15:01:08.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:01:08.496939+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:01:08.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:01:08.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 40 16 1832 1816 2.83716 0 - 0.296038 2024-04-16T15:01:09.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 41 16 1832 1816 2.76795 0 - 0.296038 2024-04-16T15:01:10.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 42 16 1832 1816 2.70205 0 - 0.296038 2024-04-16T15:01:11.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-16T15:01:11.494 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 43 16 1832 1816 2.63921 0 - 0.296038 2024-04-16T15:01:12.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-04-16T15:01:12.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:01:12.401 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T15:01:12.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T15:01:12.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 44 16 1832 1816 2.57922 0 - 0.296038 2024-04-16T15:01:12.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:12.524+0000 7fbb740ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:12.533+0000 7fbb740ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:13.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 45 16 1832 1816 2.52191 0 - 0.296038 2024-04-16T15:01:14.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 46 16 1832 1816 2.46708 0 - 0.296038 2024-04-16T15:01:15.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 47 16 1832 1816 2.41459 0 - 0.296038 2024-04-16T15:01:16.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 48 16 1832 1816 2.36429 0 - 0.296038 2024-04-16T15:01:16.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-16T15:01:17.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 49 16 1832 1816 2.31604 0 - 0.296038 2024-04-16T15:01:18.495 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 50 16 1832 1816 2.26972 0 - 0.296038 2024-04-16T15:01:19.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 51 16 1832 1816 2.22521 0 - 0.296038 2024-04-16T15:01:20.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 52 16 1832 1816 2.18242 0 - 0.296038 2024-04-16T15:01:21.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 53 16 1832 1816 2.14124 0 - 0.296038 2024-04-16T15:01:22.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-04-16T15:01:22.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 54 16 1832 1816 2.10159 0 - 0.296038 2024-04-16T15:01:23.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 55 16 1832 1816 2.06337 0 - 0.296038 2024-04-16T15:01:24.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 56 16 1832 1816 2.02653 0 - 0.296038 2024-04-16T15:01:25.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 57 16 1832 1816 1.99097 0 - 0.296038 2024-04-16T15:01:26.496 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 58 16 1832 1816 1.95665 0 - 0.296038 2024-04-16T15:01:27.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-04-16T15:01:27.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 59 16 1832 1816 1.92348 0 - 0.296038 2024-04-16T15:01:28.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:01:28.499777+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:01:28.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:01:28.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 60 16 1832 1816 1.89142 0 - 0.296038 2024-04-16T15:01:29.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 61 16 1832 1816 1.86042 0 - 0.296038 2024-04-16T15:01:30.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 62 16 1832 1816 1.83041 0 - 0.296038 2024-04-16T15:01:31.497 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 63 16 1832 1816 1.80136 0 - 0.296038 2024-04-16T15:01:32.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 64 16 1832 1816 1.77321 0 - 0.296038 2024-04-16T15:01:32.830 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2024-04-16T15:01:33.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 65 16 1832 1816 1.74593 0 - 0.296038 2024-04-16T15:01:34.045 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T15:01:34.045 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T15:01:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:01:34.050 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:01:34.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T15:01:34.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:34.180+0000 7f7e5c0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:34.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:01:34.189+0000 7f7e5c0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:01:34.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 66 16 1832 1816 1.71947 0 - 0.296038 2024-04-16T15:01:35.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 67 16 1832 1816 1.69381 0 - 0.296038 2024-04-16T15:01:36.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 68 16 1832 1816 1.6689 0 - 0.296038 2024-04-16T15:01:37.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 69 16 1832 1816 1.64471 0 - 0.296038 2024-04-16T15:01:38.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-04-16T15:01:38.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-16T15:01:38.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-16T15:01:38.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 70 16 1832 1816 1.62122 0 - 0.296038 2024-04-16T15:01:39.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 71 16 1832 1816 1.59839 0 - 0.296038 2024-04-16T15:01:40.498 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 72 16 1832 1816 1.57619 0 - 0.296038 2024-04-16T15:01:41.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 73 16 1832 1816 1.55459 0 - 0.296038 2024-04-16T15:01:42.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 74 16 1832 1816 1.53359 0 - 0.296038 2024-04-16T15:01:43.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-04-16T15:01:43.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-16T15:01:43.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2024-04-16T15:01:43.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 75 16 1832 1816 1.51314 0 - 0.296038 2024-04-16T15:01:44.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 76 16 1832 1816 1.49323 0 - 0.296038 2024-04-16T15:01:45.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 77 16 1832 1816 1.47384 0 - 0.296038 2024-04-16T15:01:46.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 78 16 1832 1816 1.45494 0 - 0.296038 2024-04-16T15:01:47.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 79 16 1832 1816 1.43652 0 - 0.296038 2024-04-16T15:01:48.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2024-04-16T15:01:48.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2024-04-16T15:01:48.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~10s 2024-04-16T15:01:48.499 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:01:48.502411+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:01:48.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:01:48.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 80 16 1832 1816 1.41857 0 - 0.296038 2024-04-16T15:01:49.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 81 16 1832 1816 1.40105 0 - 0.296038 2024-04-16T15:01:50.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 82 16 1832 1816 1.38397 0 - 0.296038 2024-04-16T15:01:51.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 83 16 1832 1816 1.36729 0 - 0.296038 2024-04-16T15:01:52.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 84 16 1832 1816 1.35102 0 - 0.296038 2024-04-16T15:01:53.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 85 16 1832 1816 1.33512 0 - 0.296038 2024-04-16T15:01:53.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~43s 2024-04-16T15:01:53.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-16T15:01:53.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~16s 2024-04-16T15:01:54.500 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 86 16 1832 1816 1.3196 0 - 0.296038 2024-04-16T15:01:55.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 87 16 1832 1816 1.30443 0 - 0.296038 2024-04-16T15:01:56.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 88 16 1832 1816 1.2896 0 - 0.296038 2024-04-16T15:01:57.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 89 16 1832 1816 1.27511 0 - 0.296038 2024-04-16T15:01:58.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 90 16 1832 1816 1.26095 0 - 0.296038 2024-04-16T15:01:58.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2024-04-16T15:01:58.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2024-04-16T15:01:58.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~21s 2024-04-16T15:01:59.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 91 16 1832 1816 1.24709 0 - 0.296038 2024-04-16T15:02:00.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 92 16 1832 1816 1.23353 0 - 0.296038 2024-04-16T15:02:01.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 93 16 1832 1816 1.22027 0 - 0.296038 2024-04-16T15:02:02.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 94 16 1832 1816 1.20729 0 - 0.296038 2024-04-16T15:02:03.501 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 95 16 1832 1816 1.19458 0 - 0.296038 2024-04-16T15:02:04.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2024-04-16T15:02:04.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~26s 2024-04-16T15:02:04.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~26s 2024-04-16T15:02:04.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 96 16 1832 1816 1.18214 0 - 0.296038 2024-04-16T15:02:05.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 97 16 1832 1816 1.16995 0 - 0.296038 2024-04-16T15:02:06.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 98 16 1832 1816 1.15801 0 - 0.296038 2024-04-16T15:02:07.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 99 16 1832 1816 1.14631 0 - 0.296038 2024-04-16T15:02:08.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:02:08.505204+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:02:08.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:02:08.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 100 16 1832 1816 1.13485 0 - 0.296038 2024-04-16T15:02:09.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2024-04-16T15:02:09.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~31s 2024-04-16T15:02:09.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~31s 2024-04-16T15:02:09.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 101 16 1832 1816 1.12361 0 - 0.296038 2024-04-16T15:02:10.502 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 102 16 1832 1816 1.1126 0 - 0.296038 2024-04-16T15:02:11.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 103 16 1832 1816 1.1018 0 - 0.296038 2024-04-16T15:02:12.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 104 16 1832 1816 1.0912 0 - 0.296038 2024-04-16T15:02:13.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 105 16 1832 1816 1.08081 0 - 0.296038 2024-04-16T15:02:14.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~63s 2024-04-16T15:02:14.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~36s 2024-04-16T15:02:14.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~36s 2024-04-16T15:02:14.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 106 16 1832 1816 1.07061 0 - 0.296038 2024-04-16T15:02:15.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 107 16 1832 1816 1.06061 0 - 0.296038 2024-04-16T15:02:16.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 108 16 1832 1816 1.05079 0 - 0.296038 2024-04-16T15:02:17.503 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 109 16 1832 1816 1.04115 0 - 0.296038 2024-04-16T15:02:18.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 110 16 1832 1816 1.03168 0 - 0.296038 2024-04-16T15:02:19.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 111 16 1832 1816 1.02239 0 - 0.296038 2024-04-16T15:02:19.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2024-04-16T15:02:19.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2024-04-16T15:02:19.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~42s 2024-04-16T15:02:20.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 112 16 1832 1816 1.01326 0 - 0.296038 2024-04-16T15:02:21.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 113 16 1832 1816 1.00429 0 - 0.296038 2024-04-16T15:02:22.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 114 16 1832 1816 0.995483 0 - 0.296038 2024-04-16T15:02:23.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 115 16 1832 1816 0.986826 0 - 0.296038 2024-04-16T15:02:24.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 116 16 1832 1816 0.978319 0 - 0.296038 2024-04-16T15:02:24.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~74s 2024-04-16T15:02:24.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~47s 2024-04-16T15:02:24.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~47s 2024-04-16T15:02:25.504 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 117 16 1832 1816 0.969958 0 - 0.296038 2024-04-16T15:02:26.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 118 16 1832 1816 0.961738 0 - 0.296038 2024-04-16T15:02:27.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 119 16 1832 1816 0.953656 0 - 0.296038 2024-04-16T15:02:28.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:02:28.507822+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:02:28.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:02:28.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 120 16 1832 1816 0.945709 0 - 0.296038 2024-04-16T15:02:29.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 121 16 1832 1816 0.937894 0 - 0.296038 2024-04-16T15:02:30.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~79s 2024-04-16T15:02:30.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~52s 2024-04-16T15:02:30.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~52s 2024-04-16T15:02:30.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 122 16 1832 1816 0.930206 0 - 0.296038 2024-04-16T15:02:31.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 123 16 1832 1816 0.922644 0 - 0.296038 2024-04-16T15:02:32.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 124 16 1832 1816 0.915203 0 - 0.296038 2024-04-16T15:02:33.505 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 125 16 1832 1816 0.907881 0 - 0.296038 2024-04-16T15:02:34.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 126 16 1832 1816 0.900676 0 - 0.296038 2024-04-16T15:02:35.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~84s 2024-04-16T15:02:35.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~57s 2024-04-16T15:02:35.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~57s 2024-04-16T15:02:35.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 127 16 1832 1816 0.893584 0 - 0.296038 2024-04-16T15:02:36.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 128 16 1832 1816 0.886603 0 - 0.296038 2024-04-16T15:02:37.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 129 16 1832 1816 0.87973 0 - 0.296038 2024-04-16T15:02:38.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 130 16 1832 1816 0.872963 0 - 0.296038 2024-04-16T15:02:39.506 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 131 16 1832 1816 0.866299 0 - 0.296038 2024-04-16T15:02:40.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~89s 2024-04-16T15:02:40.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~62s 2024-04-16T15:02:40.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~62s 2024-04-16T15:02:40.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 132 16 1832 1816 0.859736 0 - 0.296038 2024-04-16T15:02:41.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 133 16 1832 1816 0.853271 0 - 0.296038 2024-04-16T15:02:42.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 134 16 1832 1816 0.846904 0 - 0.296038 2024-04-16T15:02:43.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 135 16 1832 1816 0.84063 0 - 0.296038 2024-04-16T15:02:44.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 136 16 1832 1816 0.834449 0 - 0.296038 2024-04-16T15:02:45.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 137 16 1832 1816 0.828358 0 - 0.296038 2024-04-16T15:02:45.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-04-16T15:02:45.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~68s 2024-04-16T15:02:45.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~68s 2024-04-16T15:02:46.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 138 16 1832 1816 0.822356 0 - 0.296038 2024-04-16T15:02:47.507 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 139 16 1832 1816 0.816439 0 - 0.296038 2024-04-16T15:02:48.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:02:48.510493+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:02:48.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:02:48.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 140 16 1832 1816 0.810608 0 - 0.296038 2024-04-16T15:02:49.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 141 16 1832 1816 0.804858 0 - 0.296038 2024-04-16T15:02:50.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 142 16 1832 1816 0.799191 0 - 0.296038 2024-04-16T15:02:50.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~100s 2024-04-16T15:02:50.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~73s 2024-04-16T15:02:50.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~73s 2024-04-16T15:02:51.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 143 16 1832 1816 0.793602 0 - 0.296038 2024-04-16T15:02:52.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 144 16 1832 1816 0.788091 0 - 0.296038 2024-04-16T15:02:53.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 145 16 1832 1816 0.782655 0 - 0.296038 2024-04-16T15:02:54.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 146 16 1832 1816 0.777295 0 - 0.296038 2024-04-16T15:02:55.508 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 147 16 1832 1816 0.772007 0 - 0.296038 2024-04-16T15:02:56.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~105s 2024-04-16T15:02:56.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~78s 2024-04-16T15:02:56.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~78s 2024-04-16T15:02:56.509 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 148 16 1832 1816 0.766791 0 - 0.296038 2024-04-16T15:02:57.509 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 149 16 1832 1816 0.761644 0 - 0.296038 2024-04-16T15:02:58.509 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 150 16 1832 1816 0.756567 0 - 0.296038 2024-04-16T15:02:59.509 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 151 16 1832 1816 0.751556 0 - 0.296038 2024-04-16T15:03:00.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 152 16 1832 1816 0.746612 0 - 0.296038 2024-04-16T15:03:01.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~110s 2024-04-16T15:03:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s 2024-04-16T15:03:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~83s 2024-04-16T15:03:01.509 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 153 16 1832 1816 0.741732 0 - 0.296038 2024-04-16T15:03:02.509 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 154 16 1832 1816 0.736916 0 - 0.296038 2024-04-16T15:03:03.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 155 16 1832 1816 0.732162 0 - 0.296038 2024-04-16T15:03:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:03:03.807 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:03:04.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 156 16 1832 1816 0.727468 0 - 0.296038 2024-04-16T15:03:05.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 157 16 1832 1816 0.722835 0 - 0.296038 2024-04-16T15:03:05.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:03:05.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:03:05.931+0000 7feee2a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:03:05.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:03:05.937+0000 7feee2a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:03:05.940 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T15:03:06.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~115s 2024-04-16T15:03:06.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~88s 2024-04-16T15:03:06.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~88s 2024-04-16T15:03:06.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 158 16 1832 1816 0.71826 0 - 0.296038 2024-04-16T15:03:07.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 159 16 1832 1816 0.713742 0 - 0.296038 2024-04-16T15:03:08.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:03:08.513160+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:03:08.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:03:08.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 160 16 1832 1816 0.709282 0 - 0.296038 2024-04-16T15:03:09.510 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 161 16 1832 1816 0.704876 0 - 0.296038 2024-04-16T15:03:10.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 162 16 1832 1816 0.700525 0 - 0.296038 2024-04-16T15:03:11.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 163 16 1832 1816 0.696227 0 - 0.296038 2024-04-16T15:03:11.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~121s 2024-04-16T15:03:11.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~94s 2024-04-16T15:03:11.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~94s 2024-04-16T15:03:12.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 164 16 1832 1816 0.691982 0 - 0.296038 2024-04-16T15:03:13.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 165 16 1832 1816 0.687788 0 - 0.296038 2024-04-16T15:03:14.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 166 16 1832 1816 0.683645 0 - 0.296038 2024-04-16T15:03:15.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 167 16 1832 1816 0.679551 0 - 0.296038 2024-04-16T15:03:16.511 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 168 16 1832 1816 0.675506 0 - 0.296038 2024-04-16T15:03:16.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~126s 2024-04-16T15:03:16.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~99s 2024-04-16T15:03:16.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~99s 2024-04-16T15:03:17.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 169 16 1832 1816 0.671509 0 - 0.296038 2024-04-16T15:03:18.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 170 16 1832 1816 0.667559 0 - 0.296038 2024-04-16T15:03:19.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 171 16 1832 1816 0.663655 0 - 0.296038 2024-04-16T15:03:20.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 172 16 1832 1816 0.659796 0 - 0.296038 2024-04-16T15:03:21.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 173 16 1832 1816 0.655982 0 - 0.296038 2024-04-16T15:03:22.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~131s 2024-04-16T15:03:22.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~104s 2024-04-16T15:03:22.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~104s 2024-04-16T15:03:22.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 174 16 1832 1816 0.652212 0 - 0.296038 2024-04-16T15:03:23.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 175 16 1832 1816 0.648486 0 - 0.296038 2024-04-16T15:03:24.512 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 176 16 1832 1816 0.644801 0 - 0.296038 2024-04-16T15:03:25.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 177 16 1832 1816 0.641158 0 - 0.296038 2024-04-16T15:03:26.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 178 16 1832 1816 0.637556 0 - 0.296038 2024-04-16T15:03:27.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~136s 2024-04-16T15:03:27.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~109s 2024-04-16T15:03:27.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~109s 2024-04-16T15:03:27.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 179 16 1832 1816 0.633994 0 - 0.296038 2024-04-16T15:03:28.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:03:28.515931+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:03:28.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:03:28.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 180 16 1832 1816 0.630472 0 - 0.296038 2024-04-16T15:03:29.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 181 16 1832 1816 0.626989 0 - 0.296038 2024-04-16T15:03:30.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 182 16 1832 1816 0.623544 0 - 0.296038 2024-04-16T15:03:31.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 183 16 1832 1816 0.620136 0 - 0.296038 2024-04-16T15:03:32.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~141s 2024-04-16T15:03:32.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~114s 2024-04-16T15:03:32.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~114s 2024-04-16T15:03:32.513 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 184 16 1832 1816 0.616766 0 - 0.296038 2024-04-16T15:03:33.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 185 16 1832 1816 0.613432 0 - 0.296038 2024-04-16T15:03:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:03:34.089 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:03:34.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T15:03:34.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:03:34.213+0000 7ff36cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:03:34.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:03:34.222+0000 7ff36cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:03:34.226 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-16T15:03:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T15:03:34.230 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:03:34.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T15:03:34.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:03:34.340+0000 7f8425e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:03:34.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:03:34.349+0000 7f8425e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:03:34.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 186 16 1832 1816 0.610134 0 - 0.296038 2024-04-16T15:03:35.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 187 16 1832 1816 0.606871 0 - 0.296038 2024-04-16T15:03:36.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 188 16 1832 1816 0.603643 0 - 0.296038 2024-04-16T15:03:37.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 189 16 1832 1816 0.600449 0 - 0.296038 2024-04-16T15:03:37.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~147s 2024-04-16T15:03:37.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2024-04-16T15:03:37.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2024-04-16T15:03:38.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 190 16 1832 1816 0.597289 0 - 0.296038 2024-04-16T15:03:39.514 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 191 16 1832 1816 0.594162 0 - 0.296038 2024-04-16T15:03:40.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 192 16 1832 1816 0.591067 0 - 0.296038 2024-04-16T15:03:41.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 193 16 1832 1816 0.588005 0 - 0.296038 2024-04-16T15:03:42.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 194 16 1832 1816 0.584974 0 - 0.296038 2024-04-16T15:03:42.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~152s 2024-04-16T15:03:42.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2024-04-16T15:03:42.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~125s 2024-04-16T15:03:43.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 195 16 1832 1816 0.581974 0 - 0.296038 2024-04-16T15:03:44.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 196 16 1832 1816 0.579005 0 - 0.296038 2024-04-16T15:03:45.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 197 16 1832 1816 0.576066 0 - 0.296038 2024-04-16T15:03:46.515 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 198 16 1832 1816 0.573156 0 - 0.296038 2024-04-16T15:03:47.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 199 16 1832 1816 0.570276 0 - 0.296038 2024-04-16T15:03:48.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~157s 2024-04-16T15:03:48.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-04-16T15:03:48.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2024-04-16T15:03:48.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:03:48.518712+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:03:48.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:03:48.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 200 16 1832 1816 0.567424 0 - 0.296038 2024-04-16T15:03:49.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 201 16 1832 1816 0.564601 0 - 0.296038 2024-04-16T15:03:50.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 202 16 1832 1816 0.561806 0 - 0.296038 2024-04-16T15:03:51.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 203 16 1832 1816 0.559039 0 - 0.296038 2024-04-16T15:03:52.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 204 16 1832 1816 0.556299 0 - 0.296038 2024-04-16T15:03:53.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~162s 2024-04-16T15:03:53.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-04-16T15:03:53.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~135s 2024-04-16T15:03:53.516 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 205 16 1832 1816 0.553585 0 - 0.296038 2024-04-16T15:03:54.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 206 16 1832 1816 0.550898 0 - 0.296038 2024-04-16T15:03:55.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 207 16 1832 1816 0.548236 0 - 0.296038 2024-04-16T15:03:56.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 208 16 1832 1816 0.5456 0 - 0.296038 2024-04-16T15:03:57.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 209 16 1832 1816 0.54299 0 - 0.296038 2024-04-16T15:03:58.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~167s 2024-04-16T15:03:58.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-04-16T15:03:58.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~140s 2024-04-16T15:03:58.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 210 16 1832 1816 0.540404 0 - 0.296038 2024-04-16T15:03:59.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 211 16 1832 1816 0.537843 0 - 0.296038 2024-04-16T15:04:00.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 212 16 1832 1816 0.535306 0 - 0.296038 2024-04-16T15:04:01.517 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 213 16 1832 1816 0.532793 0 - 0.296038 2024-04-16T15:04:02.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 214 16 1832 1816 0.530304 0 - 0.296038 2024-04-16T15:04:03.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 215 16 1832 1816 0.527837 0 - 0.296038 2024-04-16T15:04:03.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~173s 2024-04-16T15:04:03.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2024-04-16T15:04:03.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~146s 2024-04-16T15:04:04.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 216 16 1832 1816 0.525393 0 - 0.296038 2024-04-16T15:04:05.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 217 16 1832 1816 0.522972 0 - 0.296038 2024-04-16T15:04:06.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 218 16 1832 1816 0.520573 0 - 0.296038 2024-04-16T15:04:07.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 219 16 1832 1816 0.518196 0 - 0.296038 2024-04-16T15:04:08.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:04:08.521293+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:04:08.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:04:08.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 220 16 1832 1816 0.515841 0 - 0.296038 2024-04-16T15:04:08.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~178s 2024-04-16T15:04:08.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2024-04-16T15:04:08.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~151s 2024-04-16T15:04:09.518 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 221 16 1832 1816 0.513506 0 - 0.296038 2024-04-16T15:04:10.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 222 16 1832 1816 0.511193 0 - 0.296038 2024-04-16T15:04:11.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 223 16 1832 1816 0.508901 0 - 0.296038 2024-04-16T15:04:12.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 224 16 1832 1816 0.506629 0 - 0.296038 2024-04-16T15:04:13.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 225 16 1832 1816 0.504377 0 - 0.296038 2024-04-16T15:04:14.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~183s 2024-04-16T15:04:14.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~156s 2024-04-16T15:04:14.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~156s 2024-04-16T15:04:14.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 226 16 1832 1816 0.502146 0 - 0.296038 2024-04-16T15:04:15.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 227 16 1832 1816 0.499933 0 - 0.296038 2024-04-16T15:04:16.519 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 228 16 1832 1816 0.497741 0 - 0.296038 2024-04-16T15:04:17.520 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 229 16 1832 1816 0.495567 0 - 0.296038 2024-04-16T15:04:18.520 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 230 16 1832 1816 0.493413 0 - 0.296038 2024-04-16T15:04:19.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~188s 2024-04-16T15:04:19.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~161s 2024-04-16T15:04:19.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~161s 2024-04-16T15:04:19.520 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 231 16 1832 1816 0.491277 0 - 0.296038 2024-04-16T15:04:20.520 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 232 16 1832 1816 0.489159 0 - 0.296038 2024-04-16T15:04:21.520 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 233 16 1832 1816 0.48706 0 - 0.296038 2024-04-16T15:04:22.520 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 234 16 1832 1816 0.484978 0 - 0.296038 2024-04-16T15:04:23.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 235 16 1832 1816 0.482914 0 - 0.296038 2024-04-16T15:04:24.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 236 16 1832 1816 0.480868 0 - 0.296038 2024-04-16T15:04:24.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~194s 2024-04-16T15:04:24.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~166s 2024-04-16T15:04:24.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~166s 2024-04-16T15:04:25.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 237 16 1832 1816 0.478839 0 - 0.296038 2024-04-16T15:04:26.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 238 16 1832 1816 0.476827 0 - 0.296038 2024-04-16T15:04:27.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 239 16 1832 1816 0.474832 0 - 0.296038 2024-04-16T15:04:28.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:04:28.524070+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:04:28.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:04:28.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 240 16 1832 1816 0.472854 0 - 0.296038 2024-04-16T15:04:29.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 241 16 1832 1816 0.470892 0 - 0.296038 2024-04-16T15:04:29.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~199s 2024-04-16T15:04:29.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~172s 2024-04-16T15:04:29.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~172s 2024-04-16T15:04:30.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 242 16 1832 1816 0.468946 0 - 0.296038 2024-04-16T15:04:31.521 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 243 16 1832 1816 0.467016 0 - 0.296038 2024-04-16T15:04:32.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 244 16 1832 1816 0.465102 0 - 0.296038 2024-04-16T15:04:33.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 245 16 1832 1816 0.463204 0 - 0.296038 2024-04-16T15:04:34.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 246 16 1832 1816 0.461321 0 - 0.296038 2024-04-16T15:04:34.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~204s 2024-04-16T15:04:34.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~177s 2024-04-16T15:04:34.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~177s 2024-04-16T15:04:35.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 247 16 1832 1816 0.459453 0 - 0.296038 2024-04-16T15:04:36.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 248 16 1832 1816 0.4576 0 - 0.296038 2024-04-16T15:04:37.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 249 16 1832 1816 0.455763 0 - 0.296038 2024-04-16T15:04:38.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 250 16 1832 1816 0.45394 0 - 0.296038 2024-04-16T15:04:39.522 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 251 16 1832 1816 0.452131 0 - 0.296038 2024-04-16T15:04:40.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~209s 2024-04-16T15:04:40.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2024-04-16T15:04:40.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~182s 2024-04-16T15:04:40.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 252 16 1832 1816 0.450337 0 - 0.296038 2024-04-16T15:04:41.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 253 16 1832 1816 0.448557 0 - 0.296038 2024-04-16T15:04:42.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 254 16 1832 1816 0.446791 0 - 0.296038 2024-04-16T15:04:43.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 255 16 1832 1816 0.445039 0 - 0.296038 2024-04-16T15:04:44.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 256 16 1832 1816 0.4433 0 - 0.296038 2024-04-16T15:04:45.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~214s 2024-04-16T15:04:45.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2024-04-16T15:04:45.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~187s 2024-04-16T15:04:45.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 257 16 1832 1816 0.441576 0 - 0.296038 2024-04-16T15:04:46.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 258 16 1832 1816 0.439864 0 - 0.296038 2024-04-16T15:04:47.523 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 259 16 1832 1816 0.438166 0 - 0.296038 2024-04-16T15:04:48.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:04:48.526645+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:04:48.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:04:48.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 260 16 1832 1816 0.43648 0 - 0.296038 2024-04-16T15:04:49.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 261 16 1832 1816 0.434808 0 - 0.296038 2024-04-16T15:04:50.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 262 16 1832 1816 0.433148 0 - 0.296038 2024-04-16T15:04:50.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2024-04-16T15:04:50.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~193s 2024-04-16T15:04:50.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~193s 2024-04-16T15:04:51.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 263 16 1832 1816 0.431502 0 - 0.296038 2024-04-16T15:04:52.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 264 16 1832 1816 0.429867 0 - 0.296038 2024-04-16T15:04:53.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 265 16 1832 1816 0.428245 0 - 0.296038 2024-04-16T15:04:54.524 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 266 16 1832 1816 0.426635 0 - 0.296038 2024-04-16T15:04:55.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 267 16 1832 1816 0.425037 0 - 0.296038 2024-04-16T15:04:55.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~225s 2024-04-16T15:04:55.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2024-04-16T15:04:55.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~198s 2024-04-16T15:04:56.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 268 16 1832 1816 0.423451 0 - 0.296038 2024-04-16T15:04:57.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 269 16 1832 1816 0.421877 0 - 0.296038 2024-04-16T15:04:58.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 270 16 1832 1816 0.420314 0 - 0.296038 2024-04-16T15:04:59.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 271 16 1832 1816 0.418764 0 - 0.296038 2024-04-16T15:05:00.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 272 16 1832 1816 0.417224 0 - 0.296038 2024-04-16T15:05:00.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~230s 2024-04-16T15:05:00.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~203s 2024-04-16T15:05:00.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~203s 2024-04-16T15:05:01.525 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 273 16 1832 1816 0.415696 0 - 0.296038 2024-04-16T15:05:02.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 274 16 1832 1816 0.414178 0 - 0.296038 2024-04-16T15:05:03.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 275 16 1832 1816 0.412672 0 - 0.296038 2024-04-16T15:05:04.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 276 16 1832 1816 0.411177 0 - 0.296038 2024-04-16T15:05:05.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 277 16 1832 1816 0.409693 0 - 0.296038 2024-04-16T15:05:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:05:05.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:05:06.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~235s 2024-04-16T15:05:06.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~208s 2024-04-16T15:05:06.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~208s 2024-04-16T15:05:06.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 278 16 1832 1816 0.408219 0 - 0.296038 2024-04-16T15:05:07.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 279 16 1832 1816 0.406756 0 - 0.296038 2024-04-16T15:05:07.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:05:07.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:05:07.973+0000 7f26fd06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:05:07.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:05:07.978+0000 7f26fd06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:05:07.982 INFO:teuthology.orchestra.run.smithi097.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T15:05:08.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:05:08.529326+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:05:08.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:05:08.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 280 16 1832 1816 0.405303 0 - 0.296038 2024-04-16T15:05:09.526 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 281 16 1832 1816 0.403861 0 - 0.296038 2024-04-16T15:05:10.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 282 16 1832 1816 0.402429 0 - 0.296038 2024-04-16T15:05:11.345 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~240s 2024-04-16T15:05:11.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~213s 2024-04-16T15:05:11.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~213s 2024-04-16T15:05:11.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 283 16 1832 1816 0.401007 0 - 0.296038 2024-04-16T15:05:12.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 284 16 1832 1816 0.399595 0 - 0.296038 2024-04-16T15:05:13.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 285 16 1832 1816 0.398193 0 - 0.296038 2024-04-16T15:05:14.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 286 16 1832 1816 0.3968 0 - 0.296038 2024-04-16T15:05:15.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 287 16 1832 1816 0.395418 0 - 0.296038 2024-04-16T15:05:16.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 288 16 1832 1816 0.394045 0 - 0.296038 2024-04-16T15:05:16.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~246s 2024-04-16T15:05:16.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~219s 2024-04-16T15:05:16.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~219s 2024-04-16T15:05:17.527 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 289 16 1832 1816 0.392681 0 - 0.296038 2024-04-16T15:05:18.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 290 16 1832 1816 0.391327 0 - 0.296038 2024-04-16T15:05:19.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 291 16 1832 1816 0.389982 0 - 0.296038 2024-04-16T15:05:20.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 292 16 1832 1816 0.388647 0 - 0.296038 2024-04-16T15:05:21.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 293 16 1832 1816 0.38732 0 - 0.296038 2024-04-16T15:05:21.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~251s 2024-04-16T15:05:21.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~224s 2024-04-16T15:05:21.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~224s 2024-04-16T15:05:22.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 294 16 1832 1816 0.386003 0 - 0.296038 2024-04-16T15:05:23.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 295 16 1832 1816 0.384695 0 - 0.296038 2024-04-16T15:05:24.528 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 296 16 1832 1816 0.383395 0 - 0.296038 2024-04-16T15:05:25.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 297 16 1832 1816 0.382104 0 - 0.296038 2024-04-16T15:05:26.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 298 16 1832 1816 0.380822 0 - 0.296038 2024-04-16T15:05:26.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~256s 2024-04-16T15:05:26.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~229s 2024-04-16T15:05:26.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~229s 2024-04-16T15:05:27.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 299 16 1832 1816 0.379548 0 - 0.296038 2024-04-16T15:05:28.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:05:28.532079+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:05:28.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:05:28.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 300 16 1832 1816 0.378283 0 - 0.296038 2024-04-16T15:05:29.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 301 16 1832 1816 0.377026 0 - 0.296038 2024-04-16T15:05:30.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 302 16 1832 1816 0.375778 0 - 0.296038 2024-04-16T15:05:31.529 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 303 16 1832 1816 0.374537 0 - 0.296038 2024-04-16T15:05:32.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~261s 2024-04-16T15:05:32.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~234s 2024-04-16T15:05:32.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~234s 2024-04-16T15:05:32.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 304 16 1832 1816 0.373305 0 - 0.296038 2024-04-16T15:05:33.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 305 16 1832 1816 0.372081 0 - 0.296038 2024-04-16T15:05:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:05:34.266 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T15:05:34.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 306 16 1832 1816 0.370866 0 - 0.296038 2024-04-16T15:05:35.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 307 16 1832 1816 0.369658 0 - 0.296038 2024-04-16T15:05:36.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 308 16 1832 1816 0.368457 0 - 0.296038 2024-04-16T15:05:37.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~266s 2024-04-16T15:05:37.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~239s 2024-04-16T15:05:37.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~239s 2024-04-16T15:05:37.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 309 16 1832 1816 0.367265 0 - 0.296038 2024-04-16T15:05:38.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 310 16 1832 1816 0.36608 0 - 0.296038 2024-04-16T15:05:39.270 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-04-16T15:05:39.270 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:05:39.270 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 3] 2024-04-16T15:05:39.270 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T15:05:39.271 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:05:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:05:39.271 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2024-04-16T15:05:39.272 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T15:05:39.530 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 311 16 1832 1816 0.364903 0 - 0.296038 2024-04-16T15:05:40.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 312 16 1832 1816 0.363734 0 - 0.296038 2024-04-16T15:05:41.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 313 16 1832 1816 0.362571 0 - 0.296038 2024-04-16T15:05:42.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 314 16 1832 1816 0.361417 0 - 0.296038 2024-04-16T15:05:42.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~272s 2024-04-16T15:05:42.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~245s 2024-04-16T15:05:42.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-16T15:05:43.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 315 16 1832 1816 0.36027 0 - 0.296038 2024-04-16T15:05:44.272 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [0, 1] 2024-04-16T15:05:44.273 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:05:44.273 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-16T15:05:44.273 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T15:05:44.273 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T15:05:44.277 INFO:tasks.ceph.osd.2:Started 2024-04-16T15:05:44.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:05:44.391 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:05:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:05:44.395 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:05:44.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 316 16 1832 1816 0.359129 0 - 0.296038 2024-04-16T15:05:45.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 317 16 1832 1816 0.357997 0 - 0.296038 2024-04-16T15:05:46.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 318 16 1832 1816 0.356871 0 - 0.296038 2024-04-16T15:05:47.531 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 319 16 1832 1816 0.355752 0 - 0.296038 2024-04-16T15:05:47.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~277s 2024-04-16T15:05:47.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~250s 2024-04-16T15:05:48.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:05:48.534671+0000 min lat: 0.0141565 max lat: 24.8627 avg lat: 0.296038 2024-04-16T15:05:48.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:05:48.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 320 16 1832 1816 0.35464 0 - 0.296038 2024-04-16T15:05:49.396 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:05:49.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 321 16 1832 1816 0.353535 0 - 0.296038 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "ops_in_flight": [ 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client_request", 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 15, 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.634 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.488092+0000" 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.488134+0000" 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.499141+0000" 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.499575+0000", 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PerShardState::osdmap_gate", 2024-04-16T15:05:49.635 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "OSDMapGate": { 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "epoch": 30 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.636 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "blocking_event": { 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "time_event": { 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 23, 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.8", 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 26, 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 26, 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 26 epoch_requested: 26 IntervalFlush", 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.638 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.556213+0000" 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.573746+0000", 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 2, 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 19 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.639 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 24, 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.640 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.8", 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.573449+0000" 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.641 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.578274+0000", 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 2, 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 19 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.642 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 25, 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.643 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.e", 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.574002+0000" 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.644 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.582582+0000", 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 22 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.645 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 26, 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.646 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.b", 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.594818+0000" 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.647 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599593+0000", 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 21 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.648 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 27, 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.649 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.e", 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 30, 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 30, 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 30 epoch_requested: 30 IntervalFlush", 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.650 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.651 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 28, 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.8", 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 30, 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 30, 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 30 epoch_requested: 30 IntervalFlush", 2024-04-16T15:05:49.652 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599674+0000", 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::await_map", 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 0, 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 28 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.653 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.654 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 29, 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.a", 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599790+0000", 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::await_map", 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 0, 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 29 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 30, 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.3", 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.659 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16, 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.4", 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.660 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 17, 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.5", 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 18, 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.3", 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 19, 2024-04-16T15:05:49.662 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.8", 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 20, 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.a", 2024-04-16T15:05:49.663 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 21, 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 21, 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.b", 2024-04-16T15:05:49.664 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 22, 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.e", 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.665 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777238, 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.6", 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.666 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.568098+0000" 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.568174+0000", 2024-04-16T15:05:49.667 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 16777236 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.668 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777239, 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.c", 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.669 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.568712+0000" 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.670 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.568779+0000", 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 16777237 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.671 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777240, 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.5", 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.672 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.568956+0000" 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.568991+0000", 2024-04-16T15:05:49.673 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 16777235 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.674 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777241, 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "1.0", 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.675 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.569252+0000" 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.569291+0000", 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.676 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 16777231 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.677 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777242, 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.0", 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.678 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.569487+0000" 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.569523+0000", 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.679 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 16777234 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.680 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777243, 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.6", 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.681 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777231, 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "1.0", 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777232, 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.3", 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 13, 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.684 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777233, 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.6", 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777234, 2024-04-16T15:05:49.685 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.0", 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777235, 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.686 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.5", 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777236, 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.6", 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.687 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 16777237, 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.c", 2024-04-16T15:05:49.688 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554451, 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.4", 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.689 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599223+0000" 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.690 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599345+0000", 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 33554449 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.691 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.692 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.692 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.692 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.692 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.692 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554452, 2024-04-16T15:05:49.692 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.9", 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.693 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599084+0000" 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.694 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599313+0000", 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 33554450 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.695 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "peering_event", 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554453, 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "PeeringEvent": { 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "from": "2", 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.1", 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "sent": 28, 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "requested": 28, 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "evt": "epoch_sent: 28 epoch_requested: 28 IntervalFlush", 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "events": [ 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.696 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599655+0000" 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "2024-04-16T15:05:49.599714+0000", 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "blocker": { 2024-04-16T15:05:49.697 INFO:teuthology.orchestra.run.smithi186.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: "waiting": 1, 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: "held_by_operation_id": 33554446 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: "initiated_at": "0.000000" 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.698 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554445, 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.0", 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 13, 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.699 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554446, 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.701 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.1", 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554447, 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.2", 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 13, 2024-04-16T15:05:49.702 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554448, 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "2.7", 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 13, 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.703 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554449, 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.4", 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 22, 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.704 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "pg_advance_map", 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 33554450, 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "detail": { 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "PGAdvanceMap": { 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "pgid": "3.9", 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "from": 23, 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "to": 30, 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: "do_init": false 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-16T15:05:49.705 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-16T15:05:50.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 322 16 1832 1816 0.352438 0 - 0.296038 2024-04-16T15:05:51.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 323 16 1832 1816 0.351346 0 - 0.296038 2024-04-16T15:05:52.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 324 16 1832 1816 0.350262 0 - 0.296038 2024-04-16T15:05:53.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~282s 2024-04-16T15:05:53.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2024-04-16T15:05:53.532 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 325 16 1832 1816 0.349184 0 - 0.296038 2024-04-16T15:05:54.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 326 16 1832 1816 0.348113 0 - 0.296038 2024-04-16T15:05:54.707 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-16T15:05:54.707 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:05:54.708 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T15:05:54.708 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T15:05:54.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T15:05:54.712 INFO:tasks.ceph.osd.3:Started 2024-04-16T15:05:54.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T15:05:54.826 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:05:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:05:54.830 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T15:05:55.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 327 16 1832 1816 0.347049 0 - 0.296038 2024-04-16T15:05:56.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 328 16 1832 1816 0.34599 0 - 0.296038 2024-04-16T15:05:57.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 329 16 1832 1816 0.344939 0 - 0.296038 2024-04-16T15:05:58.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 330 16 1832 1816 0.343894 0 - 0.296038 2024-04-16T15:05:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~287s 2024-04-16T15:05:58.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2024-04-16T15:05:59.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 331 16 1832 1816 0.342855 0 - 0.296038 2024-04-16T15:05:59.831 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T15:05:59.938 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-16T15:05:59.938 INFO:teuthology.orchestra.run.smithi186.stdout: "ops_in_flight": [] 2024-04-16T15:05:59.939 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-16T15:06:00.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 332 16 1832 1816 0.341822 0 - 0.296038 2024-04-16T15:06:01.533 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 333 16 1832 1816 0.340795 0 - 0.296038 2024-04-16T15:06:02.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 334 16 1832 1816 0.339775 0 - 0.296038 2024-04-16T15:06:03.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 335 16 1832 1816 0.338761 0 - 0.296038 2024-04-16T15:06:03.976 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~293s 2024-04-16T15:06:03.976 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~266s 2024-04-16T15:06:04.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 336 12 1832 1820 0.338497 0.000833333 302.44 0.957344 2024-04-16T15:06:04.949 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T15:06:04.949 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:06:04.949 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2024-04-16T15:06:04.950 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T15:06:04.950 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:06:05.019 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T15:06:05.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 337 12 1832 1820 0.337492 0 - 0.957344 2024-04-16T15:06:06.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 338 12 1832 1820 0.336494 0 - 0.957344 2024-04-16T15:06:07.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 339 12 1832 1820 0.335501 0 - 0.957344 2024-04-16T15:06:08.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:06:08.537307+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:06:08.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:06:08.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 340 12 1832 1820 0.334514 0 - 0.957344 2024-04-16T15:06:09.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~298s 2024-04-16T15:06:09.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~271s 2024-04-16T15:06:09.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 341 12 1832 1820 0.333533 0 - 0.957344 2024-04-16T15:06:10.020 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-04-16T15:06:10.020 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:06:10.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T15:06:10.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:06:10.143+0000 7fef6f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:06:10.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:06:10.151+0000 7fef6f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:06:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-16T15:06:10.336 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-16T15:06:10.348 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-16T15:06:09.419720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1523,"num_write_kb":97472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":15204352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1857,"ondisk_log_size":1857,"up":55,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":142352,"kb_used_data":17840,"kb_used_omap":0,"kb_used_meta":124480,"kb_avail":377345008,"statfs":{"total":386547056640,"available":386401288192,"internally_reserved":0,"allocated":18268160,"data_stored":276237117,"data_compressed":13104622,"data_compressed_allocated":17256448,"data_compressed_original":275857408,"omap_allocated":0,"internal_metadata":127467520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.933105"},"pg_stats":[{"pgid":"3.f","version":"20'101","reported_seq":115,"reported_epoch":20,"state":"stale+active+undersized+degraded","last_fresh":"2024-04-16T15:01:00.118789+0000","last_change":"2024-04-16T15:00:53.215841+0000","last_active":"2024-04-16T15:01:00.118789+0000","last_peered":"2024-04-16T15:01:00.118789+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:00:53.175427+0000","last_became_peered":"2024-04-16T15:00:53.175427+0000","last_unstale":"2024-04-16T15:01:34.043159+0000","last_undegraded":"2024-04-16T15:00:53.202755+0000","last_fullsized":"2024-04-16T15:00:53.049587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'138","reported_seq":163,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.495007+0000","last_change":"2024-04-16T15:05:59.523045+0000","last_active":"2024-04-16T15:05:49.548560+0000","last_peered":"2024-04-16T15:01:03.450127+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.439625+0000","last_became_peered":"2024-04-16T15:01:03.439625+0000","last_unstale":"2024-04-16T15:05:50.495007+0000","last_undegraded":"2024-04-16T15:05:50.495007+0000","last_fullsized":"2024-04-16T15:05:50.495007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043968,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'110","reported_seq":360,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-16T15:01:08.043844+0000","last_change":"2024-04-16T15:01:06.425078+0000","last_active":"2024-04-16T15:01:06.425078+0000","last_peered":"2024-04-16T15:01:05.797861+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.438677+0000","last_became_peered":"2024-04-16T15:01:03.438677+0000","last_unstale":"2024-04-16T15:01:08.043844+0000","last_undegraded":"2024-04-16T15:01:08.043844+0000","last_fullsized":"2024-04-16T15:01:08.043844+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":7040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":7208960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"20'112","reported_seq":6,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:00:53.059418+0000","last_change":"2024-04-16T15:05:59.526562+0000","last_active":"2024-04-16T15:00:26.387973+0000","last_peered":"2024-04-16T15:00:26.387973+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T15:00:53.059418+0000","last_undegraded":"2024-04-16T15:00:53.059418+0000","last_fullsized":"2024-04-16T15:00:53.059418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"20'111","reported_seq":125,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:01:03.191478+0000","last_change":"2024-04-16T15:05:59.517473+0000","last_active":"2024-04-16T15:01:02.157469+0000","last_peered":"2024-04-16T15:01:01.171974+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:00:53.157182+0000","last_became_peered":"2024-04-16T15:00:53.157182+0000","last_unstale":"2024-04-16T15:01:03.191478+0000","last_undegraded":"2024-04-16T15:01:03.191478+0000","last_fullsized":"2024-04-16T15:01:03.191478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"20'87","reported_seq":109,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2024-04-16T15:01:00.099929+0000","last_change":"2024-04-16T15:00:27.477816+0000","last_active":"2024-04-16T15:01:00.099929+0000","last_peered":"2024-04-16T15:01:00.099929+0000","last_clean":"2024-04-16T15:01:00.099929+0000","last_became_active":"2024-04-16T15:00:27.466744+0000","last_became_peered":"2024-04-16T15:00:27.466744+0000","last_unstale":"2024-04-16T15:01:34.043184+0000","last_undegraded":"2024-04-16T15:01:00.099929+0000","last_fullsized":"2024-04-16T15:01:00.099929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"20'104","reported_seq":126,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2024-04-16T15:01:00.185545+0000","last_change":"2024-04-16T15:00:27.482616+0000","last_active":"2024-04-16T15:01:00.185545+0000","last_peered":"2024-04-16T15:01:00.185545+0000","last_clean":"2024-04-16T15:01:00.185545+0000","last_became_active":"2024-04-16T15:00:27.473280+0000","last_became_peered":"2024-04-16T15:00:27.473280+0000","last_unstale":"2024-04-16T15:01:34.043184+0000","last_undegraded":"2024-04-16T15:01:00.185545+0000","last_fullsized":"2024-04-16T15:01:00.185545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'122","reported_seq":394,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-16T15:01:08.047458+0000","last_change":"2024-04-16T15:01:08.047458+0000","last_active":"2024-04-16T15:01:08.047458+0000","last_peered":"2024-04-16T15:01:07.555133+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.552324+0000","last_became_peered":"2024-04-16T15:01:03.552324+0000","last_unstale":"2024-04-16T15:01:08.047458+0000","last_undegraded":"2024-04-16T15:01:08.047458+0000","last_fullsized":"2024-04-16T15:01:08.047458+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":7798784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":180,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-04-16T15:06:02.558993+0000","last_change":"2024-04-16T15:06:01.571381+0000","last_active":"2024-04-16T15:06:02.558993+0000","last_peered":"2024-04-16T15:06:02.558993+0000","last_clean":"2024-04-16T15:01:09.361952+0000","last_became_active":"2024-04-16T15:06:01.571381+0000","last_became_peered":"2024-04-16T15:06:01.571381+0000","last_unstale":"2024-04-16T15:06:02.558993+0000","last_undegraded":"2024-04-16T15:06:02.558993+0000","last_fullsized":"2024-04-16T15:06:01.503760+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:12.336043+0000","last_clean_scrub_stamp":"2024-04-16T15:00:12.336043+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T15:01:09.057200+0000","last_change":"2024-04-16T15:00:15.462035+0000","last_active":"2024-04-16T15:01:09.057200+0000","last_peered":"2024-04-16T15:01:09.057200+0000","last_clean":"2024-04-16T15:01:09.057200+0000","last_became_active":"2024-04-16T15:00:15.451648+0000","last_became_peered":"2024-04-16T15:00:15.451648+0000","last_unstale":"2024-04-16T15:01:09.057200+0000","last_undegraded":"2024-04-16T15:01:09.057200+0000","last_fullsized":"2024-04-16T15:01:09.057200+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'118","reported_seq":152,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T15:01:09.048008+0000","last_change":"2024-04-16T15:01:03.652293+0000","last_active":"2024-04-16T15:01:09.048008+0000","last_peered":"2024-04-16T15:01:09.048008+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.642585+0000","last_became_peered":"2024-04-16T15:01:03.642585+0000","last_unstale":"2024-04-16T15:01:09.048008+0000","last_undegraded":"2024-04-16T15:01:03.266697+0000","last_fullsized":"2024-04-16T15:01:03.252559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7733248,"num_objects":118,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":7552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":118}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2024-04-16T15:00:55.498730+0000","last_change":"2024-04-16T15:00:15.468018+0000","last_active":"2024-04-16T15:00:55.498730+0000","last_peered":"2024-04-16T15:00:55.498730+0000","last_clean":"2024-04-16T15:00:55.498730+0000","last_became_active":"2024-04-16T15:00:15.457433+0000","last_became_peered":"2024-04-16T15:00:15.457433+0000","last_unstale":"2024-04-16T15:01:34.043186+0000","last_undegraded":"2024-04-16T15:00:55.498730+0000","last_fullsized":"2024-04-16T15:00:55.498730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'106","reported_seq":117,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2024-04-16T15:01:00.299719+0000","last_change":"2024-04-16T15:00:53.259709+0000","last_active":"2024-04-16T15:01:00.299719+0000","last_peered":"2024-04-16T15:01:00.299719+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:00:53.191286+0000","last_became_peered":"2024-04-16T15:00:53.191286+0000","last_unstale":"2024-04-16T15:01:00.299719+0000","last_undegraded":"2024-04-16T15:00:53.206998+0000","last_fullsized":"2024-04-16T15:00:53.072342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":6848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":107}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T15:01:09.035487+0000","last_change":"2024-04-16T15:01:02.247741+0000","last_active":"2024-04-16T15:01:09.035487+0000","last_peered":"2024-04-16T15:01:09.035487+0000","last_clean":"2024-04-16T15:01:09.035487+0000","last_became_active":"2024-04-16T15:01:02.242965+0000","last_became_peered":"2024-04-16T15:01:02.242965+0000","last_unstale":"2024-04-16T15:01:09.035487+0000","last_undegraded":"2024-04-16T15:01:09.035487+0000","last_fullsized":"2024-04-16T15:01:09.035487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'131","reported_seq":23,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T15:01:09.046616+0000","last_change":"2024-04-16T15:01:03.446334+0000","last_active":"2024-04-16T15:01:09.046616+0000","last_peered":"2024-04-16T15:01:09.046616+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.427903+0000","last_became_peered":"2024-04-16T15:01:03.427903+0000","last_unstale":"2024-04-16T15:01:09.046616+0000","last_undegraded":"2024-04-16T15:01:03.229798+0000","last_fullsized":"2024-04-16T15:01:03.222248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":-129}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":5,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.503285+0000","last_change":"2024-04-16T15:05:49.688385+0000","last_active":"2024-04-16T15:00:14.343077+0000","last_peered":"2024-04-16T15:00:14.343077+0000","last_clean":"2024-04-16T15:00:14.343077+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T15:05:50.503285+0000","last_undegraded":"2024-04-16T15:05:50.503285+0000","last_fullsized":"2024-04-16T15:05:50.503285+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"20'93","reported_seq":116,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.494253+0000","last_change":"2024-04-16T15:05:49.590242+0000","last_active":"2024-04-16T15:05:49.590242+0000","last_peered":"2024-04-16T15:01:02.241239+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:02.240241+0000","last_became_peered":"2024-04-16T15:01:02.240241+0000","last_unstale":"2024-04-16T15:05:50.494253+0000","last_undegraded":"2024-04-16T15:05:50.494253+0000","last_fullsized":"2024-04-16T15:05:50.494253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":5952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'107","reported_seq":150,"reported_epoch":27,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-16T15:01:09.038178+0000","last_change":"2024-04-16T15:01:03.580187+0000","last_active":"2024-04-16T15:01:09.038178+0000","last_peered":"2024-04-16T15:01:09.038178+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.416963+0000","last_became_peered":"2024-04-16T15:01:03.416963+0000","last_unstale":"2024-04-16T15:01:09.038178+0000","last_undegraded":"2024-04-16T15:01:03.174446+0000","last_fullsized":"2024-04-16T15:01:03.140831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":6848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,2],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1,2","objects":2},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.493500+0000","last_change":"2024-04-16T15:05:49.588709+0000","last_active":"2024-04-16T15:05:49.588709+0000","last_peered":"2024-04-16T15:01:02.270674+0000","last_clean":"2024-04-16T15:01:02.270674+0000","last_became_active":"2024-04-16T15:01:02.262495+0000","last_became_peered":"2024-04-16T15:01:02.262495+0000","last_unstale":"2024-04-16T15:05:50.493500+0000","last_undegraded":"2024-04-16T15:05:50.493500+0000","last_fullsized":"2024-04-16T15:05:50.493500+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'99","reported_seq":122,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.504857+0000","last_change":"2024-04-16T15:05:49.620806+0000","last_active":"2024-04-16T15:05:49.620806+0000","last_peered":"2024-04-16T15:01:02.363778+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:02.361993+0000","last_became_peered":"2024-04-16T15:01:02.361993+0000","last_unstale":"2024-04-16T15:05:50.504857+0000","last_undegraded":"2024-04-16T15:05:50.504857+0000","last_fullsized":"2024-04-16T15:05:50.504857+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":6336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-04-16T15:06:02.567279+0000","last_change":"2024-04-16T15:06:01.579638+0000","last_active":"2024-04-16T15:06:02.567279+0000","last_peered":"2024-04-16T15:06:02.567279+0000","last_clean":"2024-04-16T15:01:09.358687+0000","last_became_active":"2024-04-16T15:06:01.579638+0000","last_became_peered":"2024-04-16T15:06:01.579638+0000","last_unstale":"2024-04-16T15:06:02.567279+0000","last_undegraded":"2024-04-16T15:06:02.567279+0000","last_fullsized":"2024-04-16T15:06:01.518372+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"20'110","reported_seq":121,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2024-04-16T15:01:00.298962+0000","last_change":"2024-04-16T15:00:53.400283+0000","last_active":"2024-04-16T15:01:00.298962+0000","last_peered":"2024-04-16T15:01:00.298962+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:00:53.175698+0000","last_became_peered":"2024-04-16T15:00:53.175698+0000","last_unstale":"2024-04-16T15:01:00.298962+0000","last_undegraded":"2024-04-16T15:00:53.188733+0000","last_fullsized":"2024-04-16T15:00:53.053225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.505156+0000","last_change":"2024-04-16T15:05:49.620247+0000","last_active":"2024-04-16T15:05:49.620247+0000","last_peered":"2024-04-16T15:01:02.271001+0000","last_clean":"2024-04-16T15:01:02.271001+0000","last_became_active":"2024-04-16T15:01:02.257397+0000","last_became_peered":"2024-04-16T15:01:02.257397+0000","last_unstale":"2024-04-16T15:05:50.505156+0000","last_undegraded":"2024-04-16T15:05:50.505156+0000","last_fullsized":"2024-04-16T15:05:50.505156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'111","reported_seq":138,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-16T15:05:50.501548+0000","last_change":"2024-04-16T15:05:59.538096+0000","last_active":"2024-04-16T15:05:49.607326+0000","last_peered":"2024-04-16T15:01:03.607218+0000","last_clean":"2024-04-16T15:00:26.387973+0000","last_became_active":"2024-04-16T15:01:03.382233+0000","last_became_peered":"2024-04-16T15:01:03.382233+0000","last_unstale":"2024-04-16T15:05:50.501548+0000","last_undegraded":"2024-04-16T15:05:50.501548+0000","last_fullsized":"2024-04-16T15:05:50.501548+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:26.387973+0000","last_clean_scrub_stamp":"2024-04-16T15:00:26.387973+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T15:01:09.058037+0000","last_change":"2024-04-16T15:00:15.457441+0000","last_active":"2024-04-16T15:01:09.058037+0000","last_peered":"2024-04-16T15:01:09.058037+0000","last_clean":"2024-04-16T15:01:09.058037+0000","last_became_active":"2024-04-16T15:00:15.444424+0000","last_became_peered":"2024-04-16T15:00:15.444424+0000","last_unstale":"2024-04-16T15:01:09.058037+0000","last_undegraded":"2024-04-16T15:01:09.058037+0000","last_fullsized":"2024-04-16T15:01:09.058037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T15:00:14.343077+0000","last_clean_scrub_stamp":"2024-04-16T15:00:14.343077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1523,"num_write_kb":97472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":15204352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1760,"ondisk_log_size":1760,"up":35,"acting":32,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":33,"seq":141733920772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34088,"kb_used_data":2784,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94337752,"statfs":{"total":96636764160,"available":96601858048,"internally_reserved":0,"allocated":2850816,"data_stored":40673233,"data_compressed":1927872,"data_compressed_allocated":2539520,"data_compressed_original":40550400,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":30,"seq":128849018890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36500,"kb_used_data":5260,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94335340,"statfs":{"total":96636764160,"available":96599388160,"internally_reserved":0,"allocated":5386240,"data_stored":81622538,"data_compressed":3872531,"data_compressed_allocated":5099520,"data_compressed_original":81510400,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36856,"kb_used_data":5872,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94334984,"statfs":{"total":96636764160,"available":96599023616,"internally_reserved":0,"allocated":6012928,"data_stored":92679117,"data_compressed":4396401,"data_compressed_allocated":5787648,"data_compressed_original":92602368,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34908,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94336932,"statfs":{"total":96636764160,"available":96601018368,"internally_reserved":0,"allocated":4018176,"data_stored":61262229,"data_compressed":2907818,"data_compressed_allocated":3829760,"data_compressed_original":61194240,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T15:06:10.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.f 3.1 3.5 2024-04-16T15:06:10.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:06:10.459+0000 7f1d0d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:06:10.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:06:10.470+0000 7f1d0d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:06:10.534 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 342 12 1832 1820 0.332558 0 - 0.957344 2024-04-16T15:06:10.653 INFO:teuthology.orchestra.run.smithi097.stderr:instructing pg(s) [3.1,3.5,3.f] on osd.1 to force-backfill; 2024-04-16T15:06:10.654 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-16T15:06:11.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 343 12 1832 1820 0.331588 0 - 0.957344 2024-04-16T15:06:12.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 344 12 1832 1820 0.330625 0 - 0.957344 2024-04-16T15:06:13.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 345 12 1832 1820 0.329666 0 - 0.957344 2024-04-16T15:06:14.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 346 12 1832 1820 0.328713 0 - 0.957344 2024-04-16T15:06:14.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2024-04-16T15:06:14.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-04-16T15:06:14.583 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T15:06:14.886 INFO:tasks.ceph.osd.3:Sent signal 15 2024-04-16T15:06:14.887 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-16T15:06:14.887 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-16T15:06:14.887 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-16T15:06:14.890+0000 7fe5327ff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29727) UID: 0 2024-04-16T15:06:14.887 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-16T15:06:14.890+0000 7fe5327ff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T15:06:14.888 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:06:14.889+0000 7f248adf7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29789) UID: 0 2024-04-16T15:06:14.888 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-16T15:06:14.889+0000 7f248adf7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-16T15:06:15.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 347 12 1832 1820 0.327766 0 - 0.957344 2024-04-16T15:06:15.667 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-04-16T15:06:15.667 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:06:15.668 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 3] 2024-04-16T15:06:15.668 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T15:06:15.668 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:06:15.735 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T15:06:16.535 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 348 12 1832 1820 0.326824 0 - 0.957344 2024-04-16T15:06:17.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 349 12 1832 1820 0.325888 0 - 0.957344 2024-04-16T15:06:18.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 350 12 1832 1820 0.324957 0 - 0.957344 2024-04-16T15:06:19.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 351 12 1832 1820 0.324031 0 - 0.957344 2024-04-16T15:06:20.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 352 12 1832 1820 0.32311 0 - 0.957344 2024-04-16T15:06:20.735 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [0, 1] 2024-04-16T15:06:20.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T15:06:20.736 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-16T15:06:20.736 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T15:06:20.736 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T15:06:20.739 INFO:tasks.ceph.osd.2:Started 2024-04-16T15:06:20.739 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:20.854 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:20.857 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:21.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 353 12 1832 1820 0.322195 0 - 0.957344 2024-04-16T15:06:22.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 354 12 1832 1820 0.321285 0 - 0.957344 2024-04-16T15:06:23.536 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 355 12 1832 1820 0.32038 0 - 0.957344 2024-04-16T15:06:24.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 356 12 1832 1820 0.31948 0 - 0.957344 2024-04-16T15:06:25.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 357 12 1832 1820 0.318585 0 - 0.957344 2024-04-16T15:06:25.858 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:25.974 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:25.977 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:26.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 358 12 1832 1820 0.317695 0 - 0.957344 2024-04-16T15:06:27.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 359 12 1832 1820 0.31681 0 - 0.957344 2024-04-16T15:06:28.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:06:28.540137+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:06:28.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:06:28.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 360 12 1832 1820 0.31593 0 - 0.957344 2024-04-16T15:06:29.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 361 12 1832 1820 0.315055 0 - 0.957344 2024-04-16T15:06:30.537 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 362 12 1832 1820 0.314185 0 - 0.957344 2024-04-16T15:06:30.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:31.091 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:31.095 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:31.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 363 12 1832 1820 0.313319 0 - 0.957344 2024-04-16T15:06:32.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 364 12 1832 1820 0.312458 0 - 0.957344 2024-04-16T15:06:33.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 365 12 1832 1820 0.311602 0 - 0.957344 2024-04-16T15:06:34.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 366 12 1832 1820 0.310751 0 - 0.957344 2024-04-16T15:06:35.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 367 12 1832 1820 0.309904 0 - 0.957344 2024-04-16T15:06:36.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:36.210 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:36.213 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:36.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 368 12 1832 1820 0.309062 0 - 0.957344 2024-04-16T15:06:37.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 369 12 1832 1820 0.308224 0 - 0.957344 2024-04-16T15:06:38.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 370 12 1832 1820 0.307391 0 - 0.957344 2024-04-16T15:06:39.538 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 371 12 1832 1820 0.306563 0 - 0.957344 2024-04-16T15:06:40.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 372 12 1832 1820 0.305739 0 - 0.957344 2024-04-16T15:06:41.214 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:41.327 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:41.331 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:41.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 373 12 1832 1820 0.304919 0 - 0.957344 2024-04-16T15:06:42.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 374 12 1832 1820 0.304104 0 - 0.957344 2024-04-16T15:06:43.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 375 12 1832 1820 0.303293 0 - 0.957344 2024-04-16T15:06:44.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 376 12 1832 1820 0.302486 0 - 0.957344 2024-04-16T15:06:45.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 377 12 1832 1820 0.301684 0 - 0.957344 2024-04-16T15:06:46.332 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:46.445 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:46.449 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:46.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 378 12 1832 1820 0.300886 0 - 0.957344 2024-04-16T15:06:47.539 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 379 12 1832 1820 0.300092 0 - 0.957344 2024-04-16T15:06:48.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:06:48.542746+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:06:48.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:06:48.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 380 12 1832 1820 0.299302 0 - 0.957344 2024-04-16T15:06:49.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 381 12 1832 1820 0.298517 0 - 0.957344 2024-04-16T15:06:50.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 382 12 1832 1820 0.297735 0 - 0.957344 2024-04-16T15:06:51.450 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:51.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 383 12 1832 1820 0.296958 0 - 0.957344 2024-04-16T15:06:51.564 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:51.567 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:52.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 384 12 1832 1820 0.296184 0 - 0.957344 2024-04-16T15:06:53.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 385 12 1832 1820 0.295415 0 - 0.957344 2024-04-16T15:06:54.540 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 386 12 1832 1820 0.29465 0 - 0.957344 2024-04-16T15:06:55.541 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 387 12 1832 1820 0.293888 0 - 0.957344 2024-04-16T15:06:56.541 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 388 12 1832 1820 0.293131 0 - 0.957344 2024-04-16T15:06:56.568 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:06:56.681 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:06:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:06:56.684 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:06:57.541 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 389 12 1832 1820 0.292377 0 - 0.957344 2024-04-16T15:06:58.541 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 390 12 1832 1820 0.291628 0 - 0.957344 2024-04-16T15:06:59.541 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 391 12 1832 1820 0.290882 0 - 0.957344 2024-04-16T15:07:00.541 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 392 12 1832 1820 0.29014 0 - 0.957344 2024-04-16T15:07:01.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 393 12 1832 1820 0.289402 0 - 0.957344 2024-04-16T15:07:01.686 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:01.798 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:01.802 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:02.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 394 12 1832 1820 0.288667 0 - 0.957344 2024-04-16T15:07:03.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 395 12 1832 1820 0.287936 0 - 0.957344 2024-04-16T15:07:04.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 396 12 1832 1820 0.287209 0 - 0.957344 2024-04-16T15:07:05.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 397 12 1832 1820 0.286486 0 - 0.957344 2024-04-16T15:07:06.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 398 12 1832 1820 0.285766 0 - 0.957344 2024-04-16T15:07:06.803 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:06.917 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:06.921 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:07.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 399 12 1832 1820 0.28505 0 - 0.957344 2024-04-16T15:07:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:07:07.891 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:07:08.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:07:08.545469+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:07:08.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:07:08.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 400 12 1832 1820 0.284337 0 - 0.957344 2024-04-16T15:07:09.542 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 401 12 1832 1820 0.283628 0 - 0.957344 2024-04-16T15:07:09.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:07:10.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:07:10.014+0000 7fe07995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:07:10.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 402 12 1832 1820 0.282922 0 - 0.957344 2024-04-16T15:07:11.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 403 12 1832 1820 0.28222 0 - 0.957344 2024-04-16T15:07:11.922 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:12.035 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:12.038 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:12.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 404 12 1832 1820 0.281522 0 - 0.957344 2024-04-16T15:07:13.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 405 12 1832 1820 0.280827 0 - 0.957344 2024-04-16T15:07:14.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 406 12 1832 1820 0.280135 0 - 0.957344 2024-04-16T15:07:15.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 407 12 1832 1820 0.279447 0 - 0.957344 2024-04-16T15:07:16.543 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 408 12 1832 1820 0.278762 0 - 0.957344 2024-04-16T15:07:17.040 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:17.154 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:17.158 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:17.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 409 12 1832 1820 0.27808 0 - 0.957344 2024-04-16T15:07:18.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 410 12 1832 1820 0.277402 0 - 0.957344 2024-04-16T15:07:19.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 411 12 1832 1820 0.276727 0 - 0.957344 2024-04-16T15:07:20.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 412 12 1832 1820 0.276055 0 - 0.957344 2024-04-16T15:07:21.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 413 12 1832 1820 0.275387 0 - 0.957344 2024-04-16T15:07:22.159 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:22.273 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:22.276 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:22.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 414 12 1832 1820 0.274722 0 - 0.957344 2024-04-16T15:07:23.544 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 415 12 1832 1820 0.27406 0 - 0.957344 2024-04-16T15:07:24.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 416 12 1832 1820 0.273401 0 - 0.957344 2024-04-16T15:07:25.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 417 12 1832 1820 0.272745 0 - 0.957344 2024-04-16T15:07:26.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 418 12 1832 1820 0.272093 0 - 0.957344 2024-04-16T15:07:27.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:27.392 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:27.395 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:27.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 419 12 1832 1820 0.271443 0 - 0.957344 2024-04-16T15:07:28.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:07:28.548223+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:07:28.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:07:28.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 420 12 1832 1820 0.270797 0 - 0.957344 2024-04-16T15:07:29.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 421 12 1832 1820 0.270154 0 - 0.957344 2024-04-16T15:07:30.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 422 12 1832 1820 0.269514 0 - 0.957344 2024-04-16T15:07:31.545 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 423 12 1832 1820 0.268877 0 - 0.957344 2024-04-16T15:07:32.396 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:32.508 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:32.512 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:32.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 424 12 1832 1820 0.268242 0 - 0.957344 2024-04-16T15:07:33.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 425 12 1832 1820 0.267611 0 - 0.957344 2024-04-16T15:07:34.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 426 12 1832 1820 0.266983 0 - 0.957344 2024-04-16T15:07:35.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 427 12 1832 1820 0.266358 0 - 0.957344 2024-04-16T15:07:36.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 428 12 1832 1820 0.265736 0 - 0.957344 2024-04-16T15:07:37.513 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:37.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 429 12 1832 1820 0.265116 0 - 0.957344 2024-04-16T15:07:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:37.629 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:38.546 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 430 12 1832 1820 0.2645 0 - 0.957344 2024-04-16T15:07:39.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 431 12 1832 1820 0.263886 0 - 0.957344 2024-04-16T15:07:40.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 432 12 1832 1820 0.263275 0 - 0.957344 2024-04-16T15:07:41.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 433 12 1832 1820 0.262667 0 - 0.957344 2024-04-16T15:07:42.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 434 12 1832 1820 0.262062 0 - 0.957344 2024-04-16T15:07:42.630 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:42.743 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:42.746 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:43.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 435 12 1832 1820 0.261459 0 - 0.957344 2024-04-16T15:07:44.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 436 12 1832 1820 0.26086 0 - 0.957344 2024-04-16T15:07:45.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 437 12 1832 1820 0.260263 0 - 0.957344 2024-04-16T15:07:46.547 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 438 12 1832 1820 0.259669 0 - 0.957344 2024-04-16T15:07:47.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 439 12 1832 1820 0.259077 0 - 0.957344 2024-04-16T15:07:47.746 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:47.859 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:47.863 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:48.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:07:48.550790+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:07:48.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:07:48.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 440 12 1832 1820 0.258488 0 - 0.957344 2024-04-16T15:07:49.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 441 12 1832 1820 0.257902 0 - 0.957344 2024-04-16T15:07:50.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 442 12 1832 1820 0.257319 0 - 0.957344 2024-04-16T15:07:51.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 443 12 1832 1820 0.256738 0 - 0.957344 2024-04-16T15:07:52.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 444 12 1832 1820 0.25616 0 - 0.957344 2024-04-16T15:07:52.864 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:52.977 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:52.981 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:53.548 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 445 12 1832 1820 0.255584 0 - 0.957344 2024-04-16T15:07:54.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 446 12 1832 1820 0.255011 0 - 0.957344 2024-04-16T15:07:55.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 447 12 1832 1820 0.25444 0 - 0.957344 2024-04-16T15:07:56.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 448 12 1832 1820 0.253872 0 - 0.957344 2024-04-16T15:07:57.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 449 12 1832 1820 0.253307 0 - 0.957344 2024-04-16T15:07:57.982 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:07:58.097 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:07:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:07:58.100 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:07:58.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 450 12 1832 1820 0.252744 0 - 0.957344 2024-04-16T15:07:59.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 451 12 1832 1820 0.252184 0 - 0.957344 2024-04-16T15:08:00.549 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 452 12 1832 1820 0.251626 0 - 0.957344 2024-04-16T15:08:01.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 453 12 1832 1820 0.25107 0 - 0.957344 2024-04-16T15:08:02.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 454 12 1832 1820 0.250517 0 - 0.957344 2024-04-16T15:08:03.101 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:03.215 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:03.218 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:03.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 455 12 1832 1820 0.249967 0 - 0.957344 2024-04-16T15:08:04.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 456 12 1832 1820 0.249418 0 - 0.957344 2024-04-16T15:08:05.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 457 12 1832 1820 0.248873 0 - 0.957344 2024-04-16T15:08:06.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 458 12 1832 1820 0.248329 0 - 0.957344 2024-04-16T15:08:07.550 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 459 12 1832 1820 0.247788 0 - 0.957344 2024-04-16T15:08:08.219 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:08.332 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:08.336 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:08.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:08:08.553574+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:08:08.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:08:08.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 460 12 1832 1820 0.24725 0 - 0.957344 2024-04-16T15:08:09.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 461 12 1832 1820 0.246713 0 - 0.957344 2024-04-16T15:08:10.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 462 12 1832 1820 0.246179 0 - 0.957344 2024-04-16T15:08:11.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 463 12 1832 1820 0.245647 0 - 0.957344 2024-04-16T15:08:12.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 464 12 1832 1820 0.245118 0 - 0.957344 2024-04-16T15:08:13.337 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:13.452 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:13.457 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:13.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 465 12 1832 1820 0.244591 0 - 0.957344 2024-04-16T15:08:14.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 466 12 1832 1820 0.244066 0 - 0.957344 2024-04-16T15:08:15.551 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 467 12 1832 1820 0.243543 0 - 0.957344 2024-04-16T15:08:16.552 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 468 12 1832 1820 0.243023 0 - 0.957344 2024-04-16T15:08:17.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 469 12 1832 1820 0.242505 0 - 0.957344 2024-04-16T15:08:18.458 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:18.552 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 470 12 1832 1820 0.241989 0 - 0.957344 2024-04-16T15:08:18.571 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:18.574 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:19.552 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 471 12 1832 1820 0.241475 0 - 0.957344 2024-04-16T15:08:20.552 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 472 12 1832 1820 0.240963 0 - 0.957344 2024-04-16T15:08:21.552 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 473 12 1832 1820 0.240454 0 - 0.957344 2024-04-16T15:08:22.552 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 474 12 1832 1820 0.239947 0 - 0.957344 2024-04-16T15:08:23.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 475 12 1832 1820 0.239442 0 - 0.957344 2024-04-16T15:08:23.575 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:23.688 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:23.691 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:24.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 476 12 1832 1820 0.238939 0 - 0.957344 2024-04-16T15:08:25.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 477 12 1832 1820 0.238438 0 - 0.957344 2024-04-16T15:08:26.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 478 12 1832 1820 0.237939 0 - 0.957344 2024-04-16T15:08:27.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 479 12 1832 1820 0.237442 0 - 0.957344 2024-04-16T15:08:28.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:08:28.556356+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:08:28.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:08:28.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 480 12 1832 1820 0.236947 0 - 0.957344 2024-04-16T15:08:28.693 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:28.807 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:28.814 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:29.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 481 12 1832 1820 0.236455 0 - 0.957344 2024-04-16T15:08:30.553 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 482 12 1832 1820 0.235964 0 - 0.957344 2024-04-16T15:08:31.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 483 12 1832 1820 0.235476 0 - 0.957344 2024-04-16T15:08:32.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 484 12 1832 1820 0.234989 0 - 0.957344 2024-04-16T15:08:33.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 485 12 1832 1820 0.234505 0 - 0.957344 2024-04-16T15:08:33.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:33.928 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:33.932 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:34.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 486 12 1832 1820 0.234022 0 - 0.957344 2024-04-16T15:08:35.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 487 12 1832 1820 0.233542 0 - 0.957344 2024-04-16T15:08:36.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 488 12 1832 1820 0.233063 0 - 0.957344 2024-04-16T15:08:37.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 489 12 1832 1820 0.232586 0 - 0.957344 2024-04-16T15:08:38.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 490 12 1832 1820 0.232112 0 - 0.957344 2024-04-16T15:08:38.933 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:39.047 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:39.051 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:39.554 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 491 12 1832 1820 0.231639 0 - 0.957344 2024-04-16T15:08:40.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 492 12 1832 1820 0.231168 0 - 0.957344 2024-04-16T15:08:41.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 493 12 1832 1820 0.230699 0 - 0.957344 2024-04-16T15:08:42.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 494 12 1832 1820 0.230232 0 - 0.957344 2024-04-16T15:08:43.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 495 12 1832 1820 0.229767 0 - 0.957344 2024-04-16T15:08:44.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:44.165 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:44.169 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:44.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 496 12 1832 1820 0.229304 0 - 0.957344 2024-04-16T15:08:45.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 497 12 1832 1820 0.228843 0 - 0.957344 2024-04-16T15:08:46.555 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 498 12 1832 1820 0.228383 0 - 0.957344 2024-04-16T15:08:47.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 499 12 1832 1820 0.227925 0 - 0.957344 2024-04-16T15:08:48.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:08:48.558914+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:08:48.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:08:48.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 500 12 1832 1820 0.22747 0 - 0.957344 2024-04-16T15:08:49.169 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:49.283 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:49.286 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:49.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 501 12 1832 1820 0.227016 0 - 0.957344 2024-04-16T15:08:50.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 502 12 1832 1820 0.226563 0 - 0.957344 2024-04-16T15:08:51.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 503 12 1832 1820 0.226113 0 - 0.957344 2024-04-16T15:08:52.556 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 504 12 1832 1820 0.225664 0 - 0.957344 2024-04-16T15:08:53.557 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 505 12 1832 1820 0.225217 0 - 0.957344 2024-04-16T15:08:54.287 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:54.400 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:54.404 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:54.557 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 506 12 1832 1820 0.224772 0 - 0.957344 2024-04-16T15:08:55.557 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 507 12 1832 1820 0.224329 0 - 0.957344 2024-04-16T15:08:56.557 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 508 12 1832 1820 0.223887 0 - 0.957344 2024-04-16T15:08:57.557 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 509 12 1832 1820 0.223448 0 - 0.957344 2024-04-16T15:08:58.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 510 12 1832 1820 0.223009 0 - 0.957344 2024-04-16T15:08:59.405 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:08:59.517 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:08:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:08:59.521 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:08:59.557 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 511 12 1832 1820 0.222573 0 - 0.957344 2024-04-16T15:09:00.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 512 12 1832 1820 0.222138 0 - 0.957344 2024-04-16T15:09:01.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 513 12 1832 1820 0.221705 0 - 0.957344 2024-04-16T15:09:02.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 514 12 1832 1820 0.221274 0 - 0.957344 2024-04-16T15:09:03.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 515 12 1832 1820 0.220844 0 - 0.957344 2024-04-16T15:09:04.522 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:04.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 516 12 1832 1820 0.220416 0 - 0.957344 2024-04-16T15:09:04.637 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:04.641 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:05.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 517 12 1832 1820 0.21999 0 - 0.957344 2024-04-16T15:09:06.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 518 12 1832 1820 0.219565 0 - 0.957344 2024-04-16T15:09:07.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 519 12 1832 1820 0.219142 0 - 0.957344 2024-04-16T15:09:08.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:09:08.561495+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:09:08.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:09:08.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 520 12 1832 1820 0.218721 0 - 0.957344 2024-04-16T15:09:09.558 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 521 12 1832 1820 0.218301 0 - 0.957344 2024-04-16T15:09:09.642 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:09.755 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:09.759 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:09:09.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:09:10.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 522 12 1832 1820 0.217883 0 - 0.957344 2024-04-16T15:09:11.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 523 12 1832 1820 0.217466 0 - 0.957344 2024-04-16T15:09:11.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:09:12.055+0000 7f107a11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:09:12.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 524 12 1832 1820 0.217051 0 - 0.957344 2024-04-16T15:09:13.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 525 12 1832 1820 0.216638 0 - 0.957344 2024-04-16T15:09:14.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 526 12 1832 1820 0.216226 0 - 0.957344 2024-04-16T15:09:14.760 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:14.874 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:14.879 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:15.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 527 12 1832 1820 0.215816 0 - 0.957344 2024-04-16T15:09:16.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 528 12 1832 1820 0.215407 0 - 0.957344 2024-04-16T15:09:17.559 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 529 12 1832 1820 0.215 0 - 0.957344 2024-04-16T15:09:18.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 530 12 1832 1820 0.214594 0 - 0.957344 2024-04-16T15:09:19.560 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 531 12 1832 1820 0.21419 0 - 0.957344 2024-04-16T15:09:19.880 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:19.994 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:19.997 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:20.560 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 532 12 1832 1820 0.213787 0 - 0.957344 2024-04-16T15:09:21.560 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 533 12 1832 1820 0.213386 0 - 0.957344 2024-04-16T15:09:22.560 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 534 12 1832 1820 0.212987 0 - 0.957344 2024-04-16T15:09:23.560 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 535 12 1832 1820 0.212588 0 - 0.957344 2024-04-16T15:09:24.560 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 536 12 1832 1820 0.212192 0 - 0.957344 2024-04-16T15:09:24.998 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:25.114 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:25.116 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:25.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 537 12 1832 1820 0.211797 0 - 0.957344 2024-04-16T15:09:26.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 538 12 1832 1820 0.211403 0 - 0.957344 2024-04-16T15:09:27.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 539 12 1832 1820 0.211011 0 - 0.957344 2024-04-16T15:09:28.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:09:28.564070+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:09:28.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:09:28.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 540 12 1832 1820 0.21062 0 - 0.957344 2024-04-16T15:09:29.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 541 12 1832 1820 0.210231 0 - 0.957344 2024-04-16T15:09:30.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:30.229 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:30.232 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:30.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 542 12 1832 1820 0.209843 0 - 0.957344 2024-04-16T15:09:31.561 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 543 12 1832 1820 0.209456 0 - 0.957344 2024-04-16T15:09:32.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 544 12 1832 1820 0.209071 0 - 0.957344 2024-04-16T15:09:33.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 545 12 1832 1820 0.208688 0 - 0.957344 2024-04-16T15:09:34.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 546 12 1832 1820 0.208306 0 - 0.957344 2024-04-16T15:09:35.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:35.347 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:35.350 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:35.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 547 12 1832 1820 0.207925 0 - 0.957344 2024-04-16T15:09:36.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 548 12 1832 1820 0.207545 0 - 0.957344 2024-04-16T15:09:37.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 549 12 1832 1820 0.207167 0 - 0.957344 2024-04-16T15:09:38.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 550 12 1832 1820 0.206791 0 - 0.957344 2024-04-16T15:09:39.562 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 551 12 1832 1820 0.206415 0 - 0.957344 2024-04-16T15:09:40.351 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:40.464 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:40.468 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:40.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 552 12 1832 1820 0.206041 0 - 0.957344 2024-04-16T15:09:41.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 553 12 1832 1820 0.205669 0 - 0.957344 2024-04-16T15:09:42.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 554 12 1832 1820 0.205298 0 - 0.957344 2024-04-16T15:09:43.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 555 12 1832 1820 0.204928 0 - 0.957344 2024-04-16T15:09:44.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 556 12 1832 1820 0.204559 0 - 0.957344 2024-04-16T15:09:45.469 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:45.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 557 12 1832 1820 0.204192 0 - 0.957344 2024-04-16T15:09:45.583 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:45.586 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:46.563 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 558 12 1832 1820 0.203826 0 - 0.957344 2024-04-16T15:09:47.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 559 12 1832 1820 0.203461 0 - 0.957344 2024-04-16T15:09:48.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:09:48.566788+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:09:48.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:09:48.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 560 12 1832 1820 0.203098 0 - 0.957344 2024-04-16T15:09:49.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 561 12 1832 1820 0.202736 0 - 0.957344 2024-04-16T15:09:50.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 562 12 1832 1820 0.202375 0 - 0.957344 2024-04-16T15:09:50.588 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:50.701 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:50.704 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:51.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 563 12 1832 1820 0.202016 0 - 0.957344 2024-04-16T15:09:52.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 564 12 1832 1820 0.201657 0 - 0.957344 2024-04-16T15:09:53.564 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 565 12 1832 1820 0.201301 0 - 0.957344 2024-04-16T15:09:54.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 566 12 1832 1820 0.200945 0 - 0.957344 2024-04-16T15:09:55.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 567 12 1832 1820 0.200591 0 - 0.957344 2024-04-16T15:09:55.705 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:09:55.820 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:09:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:09:55.823 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:09:56.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 568 12 1832 1820 0.200237 0 - 0.957344 2024-04-16T15:09:57.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 569 12 1832 1820 0.199885 0 - 0.957344 2024-04-16T15:09:58.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 570 12 1832 1820 0.199535 0 - 0.957344 2024-04-16T15:09:59.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 571 12 1832 1820 0.199185 0 - 0.957344 2024-04-16T15:10:00.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 572 12 1832 1820 0.198837 0 - 0.957344 2024-04-16T15:10:00.824 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:00.938 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:00.941 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:01.565 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 573 12 1832 1820 0.19849 0 - 0.957344 2024-04-16T15:10:02.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 574 12 1832 1820 0.198144 0 - 0.957344 2024-04-16T15:10:03.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 575 12 1832 1820 0.1978 0 - 0.957344 2024-04-16T15:10:04.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 576 12 1832 1820 0.197456 0 - 0.957344 2024-04-16T15:10:05.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 577 12 1832 1820 0.197114 0 - 0.957344 2024-04-16T15:10:05.942 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:06.055 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:06.059 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:06.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 578 12 1832 1820 0.196773 0 - 0.957344 2024-04-16T15:10:07.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 579 12 1832 1820 0.196433 0 - 0.957344 2024-04-16T15:10:08.566 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:10:08.569519+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:10:08.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:10:08.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 580 12 1832 1820 0.196095 0 - 0.957344 2024-04-16T15:10:09.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 581 12 1832 1820 0.195757 0 - 0.957344 2024-04-16T15:10:10.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 582 12 1832 1820 0.195421 0 - 0.957344 2024-04-16T15:10:11.060 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:11.173 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:11.176 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:11.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 583 12 1832 1820 0.195085 0 - 0.957344 2024-04-16T15:10:12.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 584 12 1832 1820 0.194751 0 - 0.957344 2024-04-16T15:10:13.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 585 12 1832 1820 0.194418 0 - 0.957344 2024-04-16T15:10:14.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 586 12 1832 1820 0.194087 0 - 0.957344 2024-04-16T15:10:15.567 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 587 12 1832 1820 0.193756 0 - 0.957344 2024-04-16T15:10:16.177 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:16.291 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:16.294 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:16.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 588 12 1832 1820 0.193427 0 - 0.957344 2024-04-16T15:10:17.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 589 12 1832 1820 0.193098 0 - 0.957344 2024-04-16T15:10:18.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 590 12 1832 1820 0.192771 0 - 0.957344 2024-04-16T15:10:19.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 591 12 1832 1820 0.192445 0 - 0.957344 2024-04-16T15:10:20.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 592 12 1832 1820 0.19212 0 - 0.957344 2024-04-16T15:10:21.295 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:21.414 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:21.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 593 12 1832 1820 0.191796 0 - 0.957344 2024-04-16T15:10:22.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 594 12 1832 1820 0.191473 0 - 0.957344 2024-04-16T15:10:23.568 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 595 12 1832 1820 0.191151 0 - 0.957344 2024-04-16T15:10:24.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 596 12 1832 1820 0.19083 0 - 0.957344 2024-04-16T15:10:25.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 597 12 1832 1820 0.190511 0 - 0.957344 2024-04-16T15:10:26.414 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:26.527 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:26.531 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:26.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 598 12 1832 1820 0.190192 0 - 0.957344 2024-04-16T15:10:27.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 599 12 1832 1820 0.189874 0 - 0.957344 2024-04-16T15:10:28.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:10:28.572177+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:10:28.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:10:28.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 600 12 1832 1820 0.189558 0 - 0.957344 2024-04-16T15:10:29.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 601 12 1832 1820 0.189243 0 - 0.957344 2024-04-16T15:10:30.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 602 12 1832 1820 0.188928 0 - 0.957344 2024-04-16T15:10:31.531 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:31.569 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 603 12 1832 1820 0.188615 0 - 0.957344 2024-04-16T15:10:31.646 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:31.649 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:32.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 604 12 1832 1820 0.188303 0 - 0.957344 2024-04-16T15:10:33.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 605 12 1832 1820 0.187991 0 - 0.957344 2024-04-16T15:10:34.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 606 12 1832 1820 0.187681 0 - 0.957344 2024-04-16T15:10:35.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 607 12 1832 1820 0.187372 0 - 0.957344 2024-04-16T15:10:36.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 608 12 1832 1820 0.187064 0 - 0.957344 2024-04-16T15:10:36.651 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:36.767 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:36.770 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:37.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 609 12 1832 1820 0.186757 0 - 0.957344 2024-04-16T15:10:38.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 610 12 1832 1820 0.186451 0 - 0.957344 2024-04-16T15:10:39.570 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 611 12 1832 1820 0.186145 0 - 0.957344 2024-04-16T15:10:40.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 612 12 1832 1820 0.185841 0 - 0.957344 2024-04-16T15:10:41.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 613 12 1832 1820 0.185538 0 - 0.957344 2024-04-16T15:10:41.771 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:41.885 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:41.888 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:42.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 614 12 1832 1820 0.185236 0 - 0.957344 2024-04-16T15:10:43.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 615 12 1832 1820 0.184935 0 - 0.957344 2024-04-16T15:10:44.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 616 12 1832 1820 0.184634 0 - 0.957344 2024-04-16T15:10:45.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 617 12 1832 1820 0.184335 0 - 0.957344 2024-04-16T15:10:46.571 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 618 12 1832 1820 0.184037 0 - 0.957344 2024-04-16T15:10:46.889 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:47.006 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:47.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 619 12 1832 1820 0.18374 0 - 0.957344 2024-04-16T15:10:48.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:10:48.574723+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:10:48.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:10:48.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 620 12 1832 1820 0.183443 0 - 0.957344 2024-04-16T15:10:49.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 621 12 1832 1820 0.183148 0 - 0.957344 2024-04-16T15:10:50.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 622 12 1832 1820 0.182853 0 - 0.957344 2024-04-16T15:10:51.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 623 12 1832 1820 0.18256 0 - 0.957344 2024-04-16T15:10:52.007 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:52.120 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:52.124 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:52.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 624 12 1832 1820 0.182267 0 - 0.957344 2024-04-16T15:10:53.572 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 625 12 1832 1820 0.181976 0 - 0.957344 2024-04-16T15:10:54.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 626 12 1832 1820 0.181685 0 - 0.957344 2024-04-16T15:10:55.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 627 12 1832 1820 0.181395 0 - 0.957344 2024-04-16T15:10:56.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 628 12 1832 1820 0.181106 0 - 0.957344 2024-04-16T15:10:57.124 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:10:57.241 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:10:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:10:57.245 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:10:57.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 629 12 1832 1820 0.180818 0 - 0.957344 2024-04-16T15:10:58.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 630 12 1832 1820 0.180531 0 - 0.957344 2024-04-16T15:10:59.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 631 12 1832 1820 0.180245 0 - 0.957344 2024-04-16T15:11:00.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 632 12 1832 1820 0.17996 0 - 0.957344 2024-04-16T15:11:01.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 633 12 1832 1820 0.179676 0 - 0.957344 2024-04-16T15:11:02.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:02.359 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:02.363 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:02.573 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 634 12 1832 1820 0.179392 0 - 0.957344 2024-04-16T15:11:03.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 635 12 1832 1820 0.17911 0 - 0.957344 2024-04-16T15:11:04.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 636 12 1832 1820 0.178828 0 - 0.957344 2024-04-16T15:11:05.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 637 12 1832 1820 0.178548 0 - 0.957344 2024-04-16T15:11:06.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 638 12 1832 1820 0.178268 0 - 0.957344 2024-04-16T15:11:07.363 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:07.478 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:07.481 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:07.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 639 12 1832 1820 0.177989 0 - 0.957344 2024-04-16T15:11:08.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:11:08.577430+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:11:08.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:11:08.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 640 12 1832 1820 0.177711 0 - 0.957344 2024-04-16T15:11:09.574 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 641 12 1832 1820 0.177433 0 - 0.957344 2024-04-16T15:11:10.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 642 12 1832 1820 0.177157 0 - 0.957344 2024-04-16T15:11:11.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 643 12 1832 1820 0.176882 0 - 0.957344 2024-04-16T15:11:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:11:11.973 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:11:12.482 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:12.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 644 12 1832 1820 0.176607 0 - 0.957344 2024-04-16T15:11:12.595 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:12.599 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:13.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 645 12 1832 1820 0.176333 0 - 0.957344 2024-04-16T15:11:13.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:11:14.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:11:14.095+0000 7f19f895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:11:14.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 646 12 1832 1820 0.17606 0 - 0.957344 2024-04-16T15:11:15.575 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 647 12 1832 1820 0.175788 0 - 0.957344 2024-04-16T15:11:16.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 648 12 1832 1820 0.175517 0 - 0.957344 2024-04-16T15:11:17.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 649 12 1832 1820 0.175246 0 - 0.957344 2024-04-16T15:11:17.600 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:17.714 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:17.717 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:18.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 650 12 1832 1820 0.174977 0 - 0.957344 2024-04-16T15:11:19.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 651 12 1832 1820 0.174708 0 - 0.957344 2024-04-16T15:11:20.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 652 12 1832 1820 0.17444 0 - 0.957344 2024-04-16T15:11:21.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 653 12 1832 1820 0.174173 0 - 0.957344 2024-04-16T15:11:22.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 654 12 1832 1820 0.173906 0 - 0.957344 2024-04-16T15:11:22.719 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:22.832 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:22.835 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:23.576 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 655 12 1832 1820 0.173641 0 - 0.957344 2024-04-16T15:11:24.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 656 12 1832 1820 0.173376 0 - 0.957344 2024-04-16T15:11:25.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 657 12 1832 1820 0.173112 0 - 0.957344 2024-04-16T15:11:26.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 658 12 1832 1820 0.172849 0 - 0.957344 2024-04-16T15:11:27.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 659 12 1832 1820 0.172587 0 - 0.957344 2024-04-16T15:11:27.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:27.949 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:27.952 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:28.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:11:28.580108+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:11:28.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:11:28.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 660 12 1832 1820 0.172326 0 - 0.957344 2024-04-16T15:11:29.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 661 12 1832 1820 0.172065 0 - 0.957344 2024-04-16T15:11:30.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 662 12 1832 1820 0.171805 0 - 0.957344 2024-04-16T15:11:31.577 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 663 12 1832 1820 0.171546 0 - 0.957344 2024-04-16T15:11:32.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 664 12 1832 1820 0.171287 0 - 0.957344 2024-04-16T15:11:32.954 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:33.068 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:33.071 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:33.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 665 12 1832 1820 0.17103 0 - 0.957344 2024-04-16T15:11:34.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 666 12 1832 1820 0.170773 0 - 0.957344 2024-04-16T15:11:35.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 667 12 1832 1820 0.170517 0 - 0.957344 2024-04-16T15:11:36.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 668 12 1832 1820 0.170262 0 - 0.957344 2024-04-16T15:11:37.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 669 12 1832 1820 0.170007 0 - 0.957344 2024-04-16T15:11:38.073 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:38.185 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:38.189 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:38.578 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 670 12 1832 1820 0.169753 0 - 0.957344 2024-04-16T15:11:39.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 671 12 1832 1820 0.1695 0 - 0.957344 2024-04-16T15:11:40.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 672 12 1832 1820 0.169248 0 - 0.957344 2024-04-16T15:11:41.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 673 12 1832 1820 0.168997 0 - 0.957344 2024-04-16T15:11:42.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 674 12 1832 1820 0.168746 0 - 0.957344 2024-04-16T15:11:43.190 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:43.305 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:43.308 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:43.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 675 12 1832 1820 0.168496 0 - 0.957344 2024-04-16T15:11:44.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 676 12 1832 1820 0.168247 0 - 0.957344 2024-04-16T15:11:45.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 677 12 1832 1820 0.167998 0 - 0.957344 2024-04-16T15:11:46.579 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 678 12 1832 1820 0.16775 0 - 0.957344 2024-04-16T15:11:47.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 679 12 1832 1820 0.167503 0 - 0.957344 2024-04-16T15:11:48.310 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:48.424 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:48.427 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:48.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:11:48.582828+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:11:48.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:11:48.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 680 12 1832 1820 0.167257 0 - 0.957344 2024-04-16T15:11:49.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 681 12 1832 1820 0.167011 0 - 0.957344 2024-04-16T15:11:50.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 682 12 1832 1820 0.166767 0 - 0.957344 2024-04-16T15:11:51.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 683 12 1832 1820 0.166522 0 - 0.957344 2024-04-16T15:11:52.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 684 12 1832 1820 0.166279 0 - 0.957344 2024-04-16T15:11:53.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:53.541 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:53.545 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:53.580 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 685 12 1832 1820 0.166036 0 - 0.957344 2024-04-16T15:11:54.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 686 12 1832 1820 0.165794 0 - 0.957344 2024-04-16T15:11:55.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 687 12 1832 1820 0.165553 0 - 0.957344 2024-04-16T15:11:56.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 688 12 1832 1820 0.165312 0 - 0.957344 2024-04-16T15:11:57.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 689 12 1832 1820 0.165072 0 - 0.957344 2024-04-16T15:11:58.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:11:58.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 690 12 1832 1820 0.164833 0 - 0.957344 2024-04-16T15:11:58.659 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:11:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:11:58.662 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:11:59.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 691 12 1832 1820 0.164595 0 - 0.957344 2024-04-16T15:12:00.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 692 12 1832 1820 0.164357 0 - 0.957344 2024-04-16T15:12:01.581 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 693 12 1832 1820 0.164119 0 - 0.957344 2024-04-16T15:12:02.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 694 12 1832 1820 0.163883 0 - 0.957344 2024-04-16T15:12:03.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 695 12 1832 1820 0.163647 0 - 0.957344 2024-04-16T15:12:03.664 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:12:03.778 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:12:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:12:03.781 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:12:04.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 696 12 1832 1820 0.163412 0 - 0.957344 2024-04-16T15:12:05.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 697 12 1832 1820 0.163178 0 - 0.957344 2024-04-16T15:12:06.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 698 12 1832 1820 0.162944 0 - 0.957344 2024-04-16T15:12:07.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 699 12 1832 1820 0.162711 0 - 0.957344 2024-04-16T15:12:08.582 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:12:08.585578+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:12:08.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:12:08.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 700 12 1832 1820 0.162478 0 - 0.957344 2024-04-16T15:12:08.782 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:12:08.895 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:12:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:12:08.899 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:12:09.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 701 12 1832 1820 0.162247 0 - 0.957344 2024-04-16T15:12:10.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 702 12 1832 1820 0.162015 0 - 0.957344 2024-04-16T15:12:11.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 703 12 1832 1820 0.161785 0 - 0.957344 2024-04-16T15:12:12.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 704 12 1832 1820 0.161555 0 - 0.957344 2024-04-16T15:12:13.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 705 12 1832 1820 0.161326 0 - 0.957344 2024-04-16T15:12:13.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:12:14.012 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:12:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:12:14.016 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:12:14.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 706 12 1832 1820 0.161097 0 - 0.957344 2024-04-16T15:12:15.583 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 707 12 1832 1820 0.16087 0 - 0.957344 2024-04-16T15:12:16.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 708 12 1832 1820 0.160642 0 - 0.957344 2024-04-16T15:12:17.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 709 12 1832 1820 0.160416 0 - 0.957344 2024-04-16T15:12:18.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 710 12 1832 1820 0.16019 0 - 0.957344 2024-04-16T15:12:19.016 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:12:19.129 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:12:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:12:19.133 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:12:19.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 711 12 1832 1820 0.159965 0 - 0.957344 2024-04-16T15:12:20.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 712 12 1832 1820 0.15974 0 - 0.957344 2024-04-16T15:12:21.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 713 12 1832 1820 0.159516 0 - 0.957344 2024-04-16T15:12:22.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 714 12 1832 1820 0.159292 0 - 0.957344 2024-04-16T15:12:23.584 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 715 12 1832 1820 0.15907 0 - 0.957344 2024-04-16T15:12:24.134 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:12:24.247 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:12:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:12:24.251 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T15:12:24.585 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 716 12 1832 1820 0.158848 0 - 0.957344 2024-04-16T15:12:25.585 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 717 12 1832 1820 0.158626 0 - 0.957344 2024-04-16T15:12:26.585 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 718 12 1832 1820 0.158405 0 - 0.957344 2024-04-16T15:12:27.585 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 719 12 1832 1820 0.158185 0 - 0.957344 2024-04-16T15:12:28.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:12:28.588184+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:12:28.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:12:28.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 720 12 1832 1820 0.157965 0 - 0.957344 2024-04-16T15:12:29.252 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T15:12:29.365 INFO:teuthology.orchestra.run.smithi186.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T15:12:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T15:12:29.370 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2024-04-16T15:12:29.370 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2024-04-16T15:12:29.585 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 721 12 1832 1820 0.157746 0 - 0.957344 2024-04-16T15:12:30.585 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 722 12 1832 1820 0.157527 0 - 0.957344 2024-04-16T15:12:31.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 723 12 1832 1820 0.15731 0 - 0.957344 2024-04-16T15:12:32.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 724 12 1832 1820 0.157092 0 - 0.957344 2024-04-16T15:12:33.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 725 12 1832 1820 0.156876 0 - 0.957344 2024-04-16T15:12:34.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 726 12 1832 1820 0.15666 0 - 0.957344 2024-04-16T15:12:35.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 727 12 1832 1820 0.156444 0 - 0.957344 2024-04-16T15:12:36.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 728 12 1832 1820 0.156229 0 - 0.957344 2024-04-16T15:12:37.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 729 12 1832 1820 0.156015 0 - 0.957344 2024-04-16T15:12:38.586 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 730 12 1832 1820 0.155801 0 - 0.957344 2024-04-16T15:12:39.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 731 12 1832 1820 0.155588 0 - 0.957344 2024-04-16T15:12:40.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 732 12 1832 1820 0.155375 0 - 0.957344 2024-04-16T15:12:41.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 733 12 1832 1820 0.155163 0 - 0.957344 2024-04-16T15:12:42.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 734 12 1832 1820 0.154952 0 - 0.957344 2024-04-16T15:12:43.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 735 12 1832 1820 0.154741 0 - 0.957344 2024-04-16T15:12:44.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 736 12 1832 1820 0.154531 0 - 0.957344 2024-04-16T15:12:45.587 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 737 12 1832 1820 0.154321 0 - 0.957344 2024-04-16T15:12:46.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 738 12 1832 1820 0.154112 0 - 0.957344 2024-04-16T15:12:47.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 739 12 1832 1820 0.153904 0 - 0.957344 2024-04-16T15:12:48.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:12:48.590954+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:12:48.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:12:48.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 740 12 1832 1820 0.153696 0 - 0.957344 2024-04-16T15:12:49.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 741 12 1832 1820 0.153488 0 - 0.957344 2024-04-16T15:12:50.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 742 12 1832 1820 0.153281 0 - 0.957344 2024-04-16T15:12:51.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 743 12 1832 1820 0.153075 0 - 0.957344 2024-04-16T15:12:52.588 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 744 12 1832 1820 0.152869 0 - 0.957344 2024-04-16T15:12:53.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 745 12 1832 1820 0.152664 0 - 0.957344 2024-04-16T15:12:54.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 746 12 1832 1820 0.15246 0 - 0.957344 2024-04-16T15:12:55.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 747 12 1832 1820 0.152255 0 - 0.957344 2024-04-16T15:12:56.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 748 12 1832 1820 0.152052 0 - 0.957344 2024-04-16T15:12:57.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 749 12 1832 1820 0.151849 0 - 0.957344 2024-04-16T15:12:58.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 750 12 1832 1820 0.151646 0 - 0.957344 2024-04-16T15:12:59.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 751 12 1832 1820 0.151444 0 - 0.957344 2024-04-16T15:13:00.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 752 12 1832 1820 0.151243 0 - 0.957344 2024-04-16T15:13:01.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 753 12 1832 1820 0.151042 0 - 0.957344 2024-04-16T15:13:02.589 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 754 12 1832 1820 0.150842 0 - 0.957344 2024-04-16T15:13:03.590 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 755 12 1832 1820 0.150642 0 - 0.957344 2024-04-16T15:13:04.590 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 756 12 1832 1820 0.150443 0 - 0.957344 2024-04-16T15:13:05.590 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 757 12 1832 1820 0.150244 0 - 0.957344 2024-04-16T15:13:06.590 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 758 12 1832 1820 0.150046 0 - 0.957344 2024-04-16T15:13:07.590 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 759 12 1832 1820 0.149848 0 - 0.957344 2024-04-16T15:13:08.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:13:08.593561+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:13:08.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:13:08.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 760 12 1832 1820 0.149651 0 - 0.957344 2024-04-16T15:13:09.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 761 12 1832 1820 0.149454 0 - 0.957344 2024-04-16T15:13:10.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 762 12 1832 1820 0.149258 0 - 0.957344 2024-04-16T15:13:11.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 763 12 1832 1820 0.149063 0 - 0.957344 2024-04-16T15:13:12.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 764 12 1832 1820 0.148868 0 - 0.957344 2024-04-16T15:13:13.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 765 12 1832 1820 0.148673 0 - 0.957344 2024-04-16T15:13:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:13:14.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:13:14.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 766 12 1832 1820 0.148479 0 - 0.957344 2024-04-16T15:13:15.591 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 767 12 1832 1820 0.148285 0 - 0.957344 2024-04-16T15:13:16.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:13:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:13:16.136+0000 7fc866a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:13:16.592 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 768 12 1832 1820 0.148092 0 - 0.957344 2024-04-16T15:13:17.592 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 769 12 1832 1820 0.1479 0 - 0.957344 2024-04-16T15:13:18.592 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 770 12 1832 1820 0.147708 0 - 0.957344 2024-04-16T15:13:19.592 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 771 12 1832 1820 0.147516 0 - 0.957344 2024-04-16T15:13:20.592 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 772 12 1832 1820 0.147325 0 - 0.957344 2024-04-16T15:13:21.592 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 773 12 1832 1820 0.147134 0 - 0.957344 2024-04-16T15:13:22.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 774 12 1832 1820 0.146944 0 - 0.957344 2024-04-16T15:13:23.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 775 12 1832 1820 0.146755 0 - 0.957344 2024-04-16T15:13:24.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 776 12 1832 1820 0.146565 0 - 0.957344 2024-04-16T15:13:25.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 777 12 1832 1820 0.146377 0 - 0.957344 2024-04-16T15:13:26.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 778 12 1832 1820 0.146189 0 - 0.957344 2024-04-16T15:13:27.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 779 12 1832 1820 0.146001 0 - 0.957344 2024-04-16T15:13:28.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:13:28.596250+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:13:28.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:13:28.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 780 12 1832 1820 0.145814 0 - 0.957344 2024-04-16T15:13:29.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 781 12 1832 1820 0.145627 0 - 0.957344 2024-04-16T15:13:30.593 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 782 12 1832 1820 0.145441 0 - 0.957344 2024-04-16T15:13:31.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 783 12 1832 1820 0.145255 0 - 0.957344 2024-04-16T15:13:32.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 784 12 1832 1820 0.14507 0 - 0.957344 2024-04-16T15:13:33.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 785 12 1832 1820 0.144885 0 - 0.957344 2024-04-16T15:13:34.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 786 12 1832 1820 0.144701 0 - 0.957344 2024-04-16T15:13:35.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 787 12 1832 1820 0.144517 0 - 0.957344 2024-04-16T15:13:36.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 788 12 1832 1820 0.144334 0 - 0.957344 2024-04-16T15:13:37.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 789 12 1832 1820 0.144151 0 - 0.957344 2024-04-16T15:13:38.594 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 790 12 1832 1820 0.143968 0 - 0.957344 2024-04-16T15:13:39.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 791 12 1832 1820 0.143786 0 - 0.957344 2024-04-16T15:13:40.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 792 12 1832 1820 0.143605 0 - 0.957344 2024-04-16T15:13:41.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 793 12 1832 1820 0.143423 0 - 0.957344 2024-04-16T15:13:42.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 794 12 1832 1820 0.143243 0 - 0.957344 2024-04-16T15:13:43.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 795 12 1832 1820 0.143063 0 - 0.957344 2024-04-16T15:13:44.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 796 12 1832 1820 0.142883 0 - 0.957344 2024-04-16T15:13:45.595 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 797 12 1832 1820 0.142704 0 - 0.957344 2024-04-16T15:13:46.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 798 12 1832 1820 0.142525 0 - 0.957344 2024-04-16T15:13:47.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 799 12 1832 1820 0.142346 0 - 0.957344 2024-04-16T15:13:48.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:13:48.598891+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:13:48.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:13:48.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 800 12 1832 1820 0.142169 0 - 0.957344 2024-04-16T15:13:49.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 801 12 1832 1820 0.141991 0 - 0.957344 2024-04-16T15:13:50.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 802 12 1832 1820 0.141814 0 - 0.957344 2024-04-16T15:13:51.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 803 12 1832 1820 0.141637 0 - 0.957344 2024-04-16T15:13:52.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 804 12 1832 1820 0.141461 0 - 0.957344 2024-04-16T15:13:53.596 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 805 12 1832 1820 0.141285 0 - 0.957344 2024-04-16T15:13:54.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 806 12 1832 1820 0.14111 0 - 0.957344 2024-04-16T15:13:55.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 807 12 1832 1820 0.140935 0 - 0.957344 2024-04-16T15:13:56.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 808 12 1832 1820 0.140761 0 - 0.957344 2024-04-16T15:13:57.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 809 12 1832 1820 0.140587 0 - 0.957344 2024-04-16T15:13:58.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 810 12 1832 1820 0.140413 0 - 0.957344 2024-04-16T15:13:59.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 811 12 1832 1820 0.14024 0 - 0.957344 2024-04-16T15:14:00.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 812 12 1832 1820 0.140068 0 - 0.957344 2024-04-16T15:14:01.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 813 12 1832 1820 0.139895 0 - 0.957344 2024-04-16T15:14:02.597 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 814 12 1832 1820 0.139723 0 - 0.957344 2024-04-16T15:14:03.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 815 12 1832 1820 0.139552 0 - 0.957344 2024-04-16T15:14:04.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 816 12 1832 1820 0.139381 0 - 0.957344 2024-04-16T15:14:05.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 817 12 1832 1820 0.13921 0 - 0.957344 2024-04-16T15:14:06.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 818 12 1832 1820 0.13904 0 - 0.957344 2024-04-16T15:14:07.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 819 12 1832 1820 0.13887 0 - 0.957344 2024-04-16T15:14:08.598 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:14:08.601440+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:14:08.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:14:08.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 820 12 1832 1820 0.138701 0 - 0.957344 2024-04-16T15:14:09.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 821 12 1832 1820 0.138532 0 - 0.957344 2024-04-16T15:14:10.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 822 12 1832 1820 0.138364 0 - 0.957344 2024-04-16T15:14:11.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 823 12 1832 1820 0.138195 0 - 0.957344 2024-04-16T15:14:12.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 824 12 1832 1820 0.138028 0 - 0.957344 2024-04-16T15:14:13.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 825 12 1832 1820 0.13786 0 - 0.957344 2024-04-16T15:14:14.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 826 12 1832 1820 0.137693 0 - 0.957344 2024-04-16T15:14:15.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 827 12 1832 1820 0.137527 0 - 0.957344 2024-04-16T15:14:16.599 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 828 12 1832 1820 0.137361 0 - 0.957344 2024-04-16T15:14:17.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 829 12 1832 1820 0.137195 0 - 0.957344 2024-04-16T15:14:18.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 830 12 1832 1820 0.13703 0 - 0.957344 2024-04-16T15:14:19.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 831 12 1832 1820 0.136865 0 - 0.957344 2024-04-16T15:14:20.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 832 12 1832 1820 0.136701 0 - 0.957344 2024-04-16T15:14:21.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 833 12 1832 1820 0.136536 0 - 0.957344 2024-04-16T15:14:22.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 834 12 1832 1820 0.136373 0 - 0.957344 2024-04-16T15:14:23.600 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 835 12 1832 1820 0.136209 0 - 0.957344 2024-04-16T15:14:24.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 836 12 1832 1820 0.136046 0 - 0.957344 2024-04-16T15:14:25.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 837 12 1832 1820 0.135884 0 - 0.957344 2024-04-16T15:14:26.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 838 12 1832 1820 0.135722 0 - 0.957344 2024-04-16T15:14:27.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 839 12 1832 1820 0.13556 0 - 0.957344 2024-04-16T15:14:28.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:14:28.604180+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:14:28.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:14:28.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 840 12 1832 1820 0.135399 0 - 0.957344 2024-04-16T15:14:29.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 841 12 1832 1820 0.135238 0 - 0.957344 2024-04-16T15:14:30.601 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 842 12 1832 1820 0.135077 0 - 0.957344 2024-04-16T15:14:31.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 843 12 1832 1820 0.134917 0 - 0.957344 2024-04-16T15:14:32.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 844 12 1832 1820 0.134757 0 - 0.957344 2024-04-16T15:14:33.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 845 12 1832 1820 0.134597 0 - 0.957344 2024-04-16T15:14:34.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 846 12 1832 1820 0.134438 0 - 0.957344 2024-04-16T15:14:35.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 847 12 1832 1820 0.13428 0 - 0.957344 2024-04-16T15:14:36.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 848 12 1832 1820 0.134121 0 - 0.957344 2024-04-16T15:14:37.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 849 12 1832 1820 0.133963 0 - 0.957344 2024-04-16T15:14:38.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 850 12 1832 1820 0.133806 0 - 0.957344 2024-04-16T15:14:39.602 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 851 12 1832 1820 0.133648 0 - 0.957344 2024-04-16T15:14:40.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 852 12 1832 1820 0.133492 0 - 0.957344 2024-04-16T15:14:41.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 853 12 1832 1820 0.133335 0 - 0.957344 2024-04-16T15:14:42.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 854 12 1832 1820 0.133179 0 - 0.957344 2024-04-16T15:14:43.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 855 12 1832 1820 0.133023 0 - 0.957344 2024-04-16T15:14:44.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 856 12 1832 1820 0.132868 0 - 0.957344 2024-04-16T15:14:45.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 857 12 1832 1820 0.132713 0 - 0.957344 2024-04-16T15:14:46.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 858 12 1832 1820 0.132558 0 - 0.957344 2024-04-16T15:14:47.603 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 859 12 1832 1820 0.132404 0 - 0.957344 2024-04-16T15:14:48.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:14:48.606607+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:14:48.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:14:48.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 860 12 1832 1820 0.13225 0 - 0.957344 2024-04-16T15:14:49.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 861 12 1832 1820 0.132096 0 - 0.957344 2024-04-16T15:14:50.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 862 12 1832 1820 0.131943 0 - 0.957344 2024-04-16T15:14:51.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 863 12 1832 1820 0.13179 0 - 0.957344 2024-04-16T15:14:52.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 864 12 1832 1820 0.131638 0 - 0.957344 2024-04-16T15:14:53.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 865 12 1832 1820 0.131485 0 - 0.957344 2024-04-16T15:14:54.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 866 12 1832 1820 0.131334 0 - 0.957344 2024-04-16T15:14:55.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 867 12 1832 1820 0.131182 0 - 0.957344 2024-04-16T15:14:56.604 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 868 12 1832 1820 0.131031 0 - 0.957344 2024-04-16T15:14:57.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 869 12 1832 1820 0.13088 0 - 0.957344 2024-04-16T15:14:58.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 870 12 1832 1820 0.13073 0 - 0.957344 2024-04-16T15:14:59.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 871 12 1832 1820 0.13058 0 - 0.957344 2024-04-16T15:15:00.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 872 12 1832 1820 0.13043 0 - 0.957344 2024-04-16T15:15:01.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 873 12 1832 1820 0.13028 0 - 0.957344 2024-04-16T15:15:02.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 874 12 1832 1820 0.130131 0 - 0.957344 2024-04-16T15:15:03.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 875 12 1832 1820 0.129983 0 - 0.957344 2024-04-16T15:15:04.605 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 876 12 1832 1820 0.129834 0 - 0.957344 2024-04-16T15:15:05.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 877 12 1832 1820 0.129686 0 - 0.957344 2024-04-16T15:15:06.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 878 12 1832 1820 0.129539 0 - 0.957344 2024-04-16T15:15:07.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 879 12 1832 1820 0.129391 0 - 0.957344 2024-04-16T15:15:08.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:15:08.609183+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:15:08.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:15:08.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 880 12 1832 1820 0.129244 0 - 0.957344 2024-04-16T15:15:09.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 881 12 1832 1820 0.129097 0 - 0.957344 2024-04-16T15:15:10.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 882 12 1832 1820 0.128951 0 - 0.957344 2024-04-16T15:15:11.606 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 883 12 1832 1820 0.128805 0 - 0.957344 2024-04-16T15:15:12.607 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 884 12 1832 1820 0.128659 0 - 0.957344 2024-04-16T15:15:13.607 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 885 12 1832 1820 0.128514 0 - 0.957344 2024-04-16T15:15:14.607 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 886 12 1832 1820 0.128369 0 - 0.957344 2024-04-16T15:15:15.607 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 887 12 1832 1820 0.128224 0 - 0.957344 2024-04-16T15:15:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:15:16.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:15:16.607 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 888 12 1832 1820 0.12808 0 - 0.957344 2024-04-16T15:15:17.607 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 889 12 1832 1820 0.127936 0 - 0.957344 2024-04-16T15:15:18.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:15:18.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:15:18.179+0000 7fda1c754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:15:18.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 890 12 1832 1820 0.127792 0 - 0.957344 2024-04-16T15:15:19.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 891 12 1832 1820 0.127649 0 - 0.957344 2024-04-16T15:15:20.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 892 12 1832 1820 0.127505 0 - 0.957344 2024-04-16T15:15:21.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 893 12 1832 1820 0.127363 0 - 0.957344 2024-04-16T15:15:22.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 894 12 1832 1820 0.12722 0 - 0.957344 2024-04-16T15:15:23.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 895 12 1832 1820 0.127078 0 - 0.957344 2024-04-16T15:15:24.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 896 12 1832 1820 0.126936 0 - 0.957344 2024-04-16T15:15:25.608 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 897 12 1832 1820 0.126795 0 - 0.957344 2024-04-16T15:15:26.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 898 12 1832 1820 0.126654 0 - 0.957344 2024-04-16T15:15:27.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 899 12 1832 1820 0.126513 0 - 0.957344 2024-04-16T15:15:28.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:15:28.611966+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:15:28.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:15:28.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 900 12 1832 1820 0.126372 0 - 0.957344 2024-04-16T15:15:29.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 901 12 1832 1820 0.126232 0 - 0.957344 2024-04-16T15:15:30.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 902 12 1832 1820 0.126092 0 - 0.957344 2024-04-16T15:15:31.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 903 12 1832 1820 0.125952 0 - 0.957344 2024-04-16T15:15:32.609 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 904 12 1832 1820 0.125813 0 - 0.957344 2024-04-16T15:15:33.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 905 12 1832 1820 0.125674 0 - 0.957344 2024-04-16T15:15:34.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 906 12 1832 1820 0.125535 0 - 0.957344 2024-04-16T15:15:35.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 907 12 1832 1820 0.125397 0 - 0.957344 2024-04-16T15:15:36.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 908 12 1832 1820 0.125259 0 - 0.957344 2024-04-16T15:15:37.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 909 12 1832 1820 0.125121 0 - 0.957344 2024-04-16T15:15:38.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 910 12 1832 1820 0.124983 0 - 0.957344 2024-04-16T15:15:39.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 911 12 1832 1820 0.124846 0 - 0.957344 2024-04-16T15:15:40.610 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 912 12 1832 1820 0.124709 0 - 0.957344 2024-04-16T15:15:41.611 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 913 12 1832 1820 0.124573 0 - 0.957344 2024-04-16T15:15:42.611 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 914 12 1832 1820 0.124436 0 - 0.957344 2024-04-16T15:15:43.611 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 915 12 1832 1820 0.1243 0 - 0.957344 2024-04-16T15:15:44.611 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 916 12 1832 1820 0.124165 0 - 0.957344 2024-04-16T15:15:45.611 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 917 12 1832 1820 0.124029 0 - 0.957344 2024-04-16T15:15:46.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 918 12 1832 1820 0.123894 0 - 0.957344 2024-04-16T15:15:47.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 919 12 1832 1820 0.123759 0 - 0.957344 2024-04-16T15:15:48.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:15:48.614600+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:15:48.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:15:48.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 920 12 1832 1820 0.123625 0 - 0.957344 2024-04-16T15:15:49.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 921 12 1832 1820 0.123491 0 - 0.957344 2024-04-16T15:15:50.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 922 12 1832 1820 0.123357 0 - 0.957344 2024-04-16T15:15:51.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 923 12 1832 1820 0.123223 0 - 0.957344 2024-04-16T15:15:52.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 924 12 1832 1820 0.12309 0 - 0.957344 2024-04-16T15:15:53.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 925 12 1832 1820 0.122957 0 - 0.957344 2024-04-16T15:15:54.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 926 12 1832 1820 0.122824 0 - 0.957344 2024-04-16T15:15:55.612 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 927 12 1832 1820 0.122691 0 - 0.957344 2024-04-16T15:15:56.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 928 12 1832 1820 0.122559 0 - 0.957344 2024-04-16T15:15:57.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 929 12 1832 1820 0.122427 0 - 0.957344 2024-04-16T15:15:58.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 930 12 1832 1820 0.122296 0 - 0.957344 2024-04-16T15:15:59.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 931 12 1832 1820 0.122164 0 - 0.957344 2024-04-16T15:16:00.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 932 12 1832 1820 0.122033 0 - 0.957344 2024-04-16T15:16:01.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 933 12 1832 1820 0.121902 0 - 0.957344 2024-04-16T15:16:02.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 934 12 1832 1820 0.121772 0 - 0.957344 2024-04-16T15:16:03.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 935 12 1832 1820 0.121642 0 - 0.957344 2024-04-16T15:16:04.613 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 936 12 1832 1820 0.121512 0 - 0.957344 2024-04-16T15:16:05.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 937 12 1832 1820 0.121382 0 - 0.957344 2024-04-16T15:16:06.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 938 12 1832 1820 0.121253 0 - 0.957344 2024-04-16T15:16:07.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 939 12 1832 1820 0.121123 0 - 0.957344 2024-04-16T15:16:08.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:16:08.617121+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:16:08.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:16:08.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 940 12 1832 1820 0.120995 0 - 0.957344 2024-04-16T15:16:09.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 941 12 1832 1820 0.120866 0 - 0.957344 2024-04-16T15:16:10.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 942 12 1832 1820 0.120738 0 - 0.957344 2024-04-16T15:16:11.614 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 943 12 1832 1820 0.12061 0 - 0.957344 2024-04-16T15:16:12.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 944 12 1832 1820 0.120482 0 - 0.957344 2024-04-16T15:16:13.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 945 12 1832 1820 0.120354 0 - 0.957344 2024-04-16T15:16:14.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 946 12 1832 1820 0.120227 0 - 0.957344 2024-04-16T15:16:15.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 947 12 1832 1820 0.1201 0 - 0.957344 2024-04-16T15:16:16.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 948 12 1832 1820 0.119973 0 - 0.957344 2024-04-16T15:16:17.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 949 12 1832 1820 0.119847 0 - 0.957344 2024-04-16T15:16:18.615 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 950 12 1832 1820 0.119721 0 - 0.957344 2024-04-16T15:16:19.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 951 12 1832 1820 0.119595 0 - 0.957344 2024-04-16T15:16:20.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 952 12 1832 1820 0.119469 0 - 0.957344 2024-04-16T15:16:21.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 953 12 1832 1820 0.119344 0 - 0.957344 2024-04-16T15:16:22.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 954 12 1832 1820 0.119219 0 - 0.957344 2024-04-16T15:16:23.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 955 12 1832 1820 0.119094 0 - 0.957344 2024-04-16T15:16:24.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 956 12 1832 1820 0.11897 0 - 0.957344 2024-04-16T15:16:25.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 957 12 1832 1820 0.118845 0 - 0.957344 2024-04-16T15:16:26.616 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 958 12 1832 1820 0.118721 0 - 0.957344 2024-04-16T15:16:27.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 959 12 1832 1820 0.118597 0 - 0.957344 2024-04-16T15:16:28.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:16:28.619923+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:16:28.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:16:28.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 960 12 1832 1820 0.118474 0 - 0.957344 2024-04-16T15:16:29.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 961 12 1832 1820 0.118351 0 - 0.957344 2024-04-16T15:16:30.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 962 12 1832 1820 0.118227 0 - 0.957344 2024-04-16T15:16:31.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 963 12 1832 1820 0.118105 0 - 0.957344 2024-04-16T15:16:32.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 964 12 1832 1820 0.117982 0 - 0.957344 2024-04-16T15:16:33.617 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 965 12 1832 1820 0.11786 0 - 0.957344 2024-04-16T15:16:34.618 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 966 12 1832 1820 0.117738 0 - 0.957344 2024-04-16T15:16:35.618 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 967 12 1832 1820 0.117616 0 - 0.957344 2024-04-16T15:16:36.618 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 968 12 1832 1820 0.117495 0 - 0.957344 2024-04-16T15:16:37.618 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 969 12 1832 1820 0.117373 0 - 0.957344 2024-04-16T15:16:38.618 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 970 12 1832 1820 0.117252 0 - 0.957344 2024-04-16T15:16:39.618 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 971 12 1832 1820 0.117132 0 - 0.957344 2024-04-16T15:16:40.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 972 12 1832 1820 0.117011 0 - 0.957344 2024-04-16T15:16:41.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 973 12 1832 1820 0.116891 0 - 0.957344 2024-04-16T15:16:42.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 974 12 1832 1820 0.116771 0 - 0.957344 2024-04-16T15:16:43.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 975 12 1832 1820 0.116651 0 - 0.957344 2024-04-16T15:16:44.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 976 12 1832 1820 0.116532 0 - 0.957344 2024-04-16T15:16:45.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 977 12 1832 1820 0.116412 0 - 0.957344 2024-04-16T15:16:46.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 978 12 1832 1820 0.116293 0 - 0.957344 2024-04-16T15:16:47.619 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 979 12 1832 1820 0.116175 0 - 0.957344 2024-04-16T15:16:48.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:16:48.622671+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:16:48.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:16:48.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 980 12 1832 1820 0.116056 0 - 0.957344 2024-04-16T15:16:49.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 981 12 1832 1820 0.115938 0 - 0.957344 2024-04-16T15:16:50.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 982 12 1832 1820 0.11582 0 - 0.957344 2024-04-16T15:16:51.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 983 12 1832 1820 0.115702 0 - 0.957344 2024-04-16T15:16:52.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 984 12 1832 1820 0.115584 0 - 0.957344 2024-04-16T15:16:53.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 985 12 1832 1820 0.115467 0 - 0.957344 2024-04-16T15:16:54.620 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 986 12 1832 1820 0.11535 0 - 0.957344 2024-04-16T15:16:55.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 987 12 1832 1820 0.115233 0 - 0.957344 2024-04-16T15:16:56.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 988 12 1832 1820 0.115116 0 - 0.957344 2024-04-16T15:16:57.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 989 12 1832 1820 0.115 0 - 0.957344 2024-04-16T15:16:58.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 990 12 1832 1820 0.114884 0 - 0.957344 2024-04-16T15:16:59.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 991 12 1832 1820 0.114768 0 - 0.957344 2024-04-16T15:17:00.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 992 12 1832 1820 0.114652 0 - 0.957344 2024-04-16T15:17:01.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 993 12 1832 1820 0.114537 0 - 0.957344 2024-04-16T15:17:02.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 994 12 1832 1820 0.114421 0 - 0.957344 2024-04-16T15:17:03.621 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 995 12 1832 1820 0.114306 0 - 0.957344 2024-04-16T15:17:04.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 996 12 1832 1820 0.114192 0 - 0.957344 2024-04-16T15:17:05.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 997 12 1832 1820 0.114077 0 - 0.957344 2024-04-16T15:17:06.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 998 12 1832 1820 0.113963 0 - 0.957344 2024-04-16T15:17:07.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 999 12 1832 1820 0.113849 0 - 0.957344 2024-04-16T15:17:08.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:17:08.625395+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:17:08.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:17:08.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1000 12 1832 1820 0.113735 0 - 0.957344 2024-04-16T15:17:09.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1001 12 1832 1820 0.113621 0 - 0.957344 2024-04-16T15:17:10.622 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1002 12 1832 1820 0.113508 0 - 0.957344 2024-04-16T15:17:11.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1003 12 1832 1820 0.113395 0 - 0.957344 2024-04-16T15:17:12.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1004 12 1832 1820 0.113282 0 - 0.957344 2024-04-16T15:17:13.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1005 12 1832 1820 0.113169 0 - 0.957344 2024-04-16T15:17:14.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1006 12 1832 1820 0.113056 0 - 0.957344 2024-04-16T15:17:15.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1007 12 1832 1820 0.112944 0 - 0.957344 2024-04-16T15:17:16.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1008 12 1832 1820 0.112832 0 - 0.957344 2024-04-16T15:17:17.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1009 12 1832 1820 0.11272 0 - 0.957344 2024-04-16T15:17:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:17:18.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:17:18.623 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1010 12 1832 1820 0.112609 0 - 0.957344 2024-04-16T15:17:19.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1011 12 1832 1820 0.112497 0 - 0.957344 2024-04-16T15:17:20.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:17:20.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:17:20.220+0000 7fe96719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:17:20.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1012 12 1832 1820 0.112386 0 - 0.957344 2024-04-16T15:17:21.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1013 12 1832 1820 0.112275 0 - 0.957344 2024-04-16T15:17:22.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1014 12 1832 1820 0.112165 0 - 0.957344 2024-04-16T15:17:23.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1015 12 1832 1820 0.112054 0 - 0.957344 2024-04-16T15:17:24.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1016 12 1832 1820 0.111944 0 - 0.957344 2024-04-16T15:17:25.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1017 12 1832 1820 0.111834 0 - 0.957344 2024-04-16T15:17:26.624 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1018 12 1832 1820 0.111724 0 - 0.957344 2024-04-16T15:17:27.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1019 12 1832 1820 0.111614 0 - 0.957344 2024-04-16T15:17:28.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:17:28.627919+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:17:28.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:17:28.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1020 12 1832 1820 0.111505 0 - 0.957344 2024-04-16T15:17:29.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1021 12 1832 1820 0.111396 0 - 0.957344 2024-04-16T15:17:30.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1022 12 1832 1820 0.111287 0 - 0.957344 2024-04-16T15:17:31.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1023 12 1832 1820 0.111178 0 - 0.957344 2024-04-16T15:17:32.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1024 12 1832 1820 0.111069 0 - 0.957344 2024-04-16T15:17:33.625 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1025 12 1832 1820 0.110961 0 - 0.957344 2024-04-16T15:17:34.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1026 12 1832 1820 0.110853 0 - 0.957344 2024-04-16T15:17:35.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1027 12 1832 1820 0.110745 0 - 0.957344 2024-04-16T15:17:36.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1028 12 1832 1820 0.110637 0 - 0.957344 2024-04-16T15:17:37.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1029 12 1832 1820 0.11053 0 - 0.957344 2024-04-16T15:17:38.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1030 12 1832 1820 0.110422 0 - 0.957344 2024-04-16T15:17:39.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1031 12 1832 1820 0.110315 0 - 0.957344 2024-04-16T15:17:40.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1032 12 1832 1820 0.110208 0 - 0.957344 2024-04-16T15:17:41.626 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1033 12 1832 1820 0.110102 0 - 0.957344 2024-04-16T15:17:42.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1034 12 1832 1820 0.109995 0 - 0.957344 2024-04-16T15:17:43.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1035 12 1832 1820 0.109889 0 - 0.957344 2024-04-16T15:17:44.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1036 12 1832 1820 0.109783 0 - 0.957344 2024-04-16T15:17:45.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1037 12 1832 1820 0.109677 0 - 0.957344 2024-04-16T15:17:46.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1038 12 1832 1820 0.109571 0 - 0.957344 2024-04-16T15:17:47.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1039 12 1832 1820 0.109466 0 - 0.957344 2024-04-16T15:17:48.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:17:48.630492+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:17:48.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:17:48.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1040 12 1832 1820 0.10936 0 - 0.957344 2024-04-16T15:17:49.627 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1041 12 1832 1820 0.109255 0 - 0.957344 2024-04-16T15:17:50.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1042 12 1832 1820 0.109151 0 - 0.957344 2024-04-16T15:17:51.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1043 12 1832 1820 0.109046 0 - 0.957344 2024-04-16T15:17:52.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1044 12 1832 1820 0.108941 0 - 0.957344 2024-04-16T15:17:53.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1045 12 1832 1820 0.108837 0 - 0.957344 2024-04-16T15:17:54.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1046 12 1832 1820 0.108733 0 - 0.957344 2024-04-16T15:17:55.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1047 12 1832 1820 0.108629 0 - 0.957344 2024-04-16T15:17:56.628 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1048 12 1832 1820 0.108526 0 - 0.957344 2024-04-16T15:17:57.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1049 12 1832 1820 0.108422 0 - 0.957344 2024-04-16T15:17:58.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1050 12 1832 1820 0.108319 0 - 0.957344 2024-04-16T15:17:59.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1051 12 1832 1820 0.108216 0 - 0.957344 2024-04-16T15:18:00.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1052 12 1832 1820 0.108113 0 - 0.957344 2024-04-16T15:18:01.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1053 12 1832 1820 0.10801 0 - 0.957344 2024-04-16T15:18:02.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1054 12 1832 1820 0.107908 0 - 0.957344 2024-04-16T15:18:03.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1055 12 1832 1820 0.107806 0 - 0.957344 2024-04-16T15:18:04.629 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1056 12 1832 1820 0.107703 0 - 0.957344 2024-04-16T15:18:05.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1057 12 1832 1820 0.107602 0 - 0.957344 2024-04-16T15:18:06.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1058 12 1832 1820 0.1075 0 - 0.957344 2024-04-16T15:18:07.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1059 12 1832 1820 0.107398 0 - 0.957344 2024-04-16T15:18:08.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:18:08.633113+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:18:08.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:18:08.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1060 12 1832 1820 0.107297 0 - 0.957344 2024-04-16T15:18:09.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1061 12 1832 1820 0.107196 0 - 0.957344 2024-04-16T15:18:10.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1062 12 1832 1820 0.107095 0 - 0.957344 2024-04-16T15:18:11.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1063 12 1832 1820 0.106994 0 - 0.957344 2024-04-16T15:18:12.630 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1064 12 1832 1820 0.106894 0 - 0.957344 2024-04-16T15:18:13.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1065 12 1832 1820 0.106793 0 - 0.957344 2024-04-16T15:18:14.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1066 12 1832 1820 0.106693 0 - 0.957344 2024-04-16T15:18:15.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1067 12 1832 1820 0.106593 0 - 0.957344 2024-04-16T15:18:16.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1068 12 1832 1820 0.106493 0 - 0.957344 2024-04-16T15:18:17.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1069 12 1832 1820 0.106394 0 - 0.957344 2024-04-16T15:18:18.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1070 12 1832 1820 0.106294 0 - 0.957344 2024-04-16T15:18:19.631 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1071 12 1832 1820 0.106195 0 - 0.957344 2024-04-16T15:18:20.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1072 12 1832 1820 0.106096 0 - 0.957344 2024-04-16T15:18:21.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1073 12 1832 1820 0.105997 0 - 0.957344 2024-04-16T15:18:22.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1074 12 1832 1820 0.105898 0 - 0.957344 2024-04-16T15:18:23.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1075 12 1832 1820 0.1058 0 - 0.957344 2024-04-16T15:18:24.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1076 12 1832 1820 0.105702 0 - 0.957344 2024-04-16T15:18:25.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1077 12 1832 1820 0.105603 0 - 0.957344 2024-04-16T15:18:26.632 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1078 12 1832 1820 0.105505 0 - 0.957344 2024-04-16T15:18:27.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1079 12 1832 1820 0.105408 0 - 0.957344 2024-04-16T15:18:28.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:18:28.635825+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:18:28.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:18:28.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1080 12 1832 1820 0.10531 0 - 0.957344 2024-04-16T15:18:29.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1081 12 1832 1820 0.105213 0 - 0.957344 2024-04-16T15:18:30.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1082 12 1832 1820 0.105115 0 - 0.957344 2024-04-16T15:18:31.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1083 12 1832 1820 0.105018 0 - 0.957344 2024-04-16T15:18:32.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1084 12 1832 1820 0.104921 0 - 0.957344 2024-04-16T15:18:33.633 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1085 12 1832 1820 0.104825 0 - 0.957344 2024-04-16T15:18:34.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1086 12 1832 1820 0.104728 0 - 0.957344 2024-04-16T15:18:35.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1087 12 1832 1820 0.104632 0 - 0.957344 2024-04-16T15:18:36.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1088 12 1832 1820 0.104536 0 - 0.957344 2024-04-16T15:18:37.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1089 12 1832 1820 0.10444 0 - 0.957344 2024-04-16T15:18:38.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1090 12 1832 1820 0.104344 0 - 0.957344 2024-04-16T15:18:39.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1091 12 1832 1820 0.104248 0 - 0.957344 2024-04-16T15:18:40.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1092 12 1832 1820 0.104153 0 - 0.957344 2024-04-16T15:18:41.634 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1093 12 1832 1820 0.104058 0 - 0.957344 2024-04-16T15:18:42.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1094 12 1832 1820 0.103962 0 - 0.957344 2024-04-16T15:18:43.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1095 12 1832 1820 0.103867 0 - 0.957344 2024-04-16T15:18:44.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1096 12 1832 1820 0.103773 0 - 0.957344 2024-04-16T15:18:45.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1097 12 1832 1820 0.103678 0 - 0.957344 2024-04-16T15:18:46.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1098 12 1832 1820 0.103584 0 - 0.957344 2024-04-16T15:18:47.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1099 12 1832 1820 0.103489 0 - 0.957344 2024-04-16T15:18:48.635 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:18:48.638590+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:18:48.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:18:48.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1100 12 1832 1820 0.103395 0 - 0.957344 2024-04-16T15:18:49.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1101 12 1832 1820 0.103301 0 - 0.957344 2024-04-16T15:18:50.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1102 12 1832 1820 0.103208 0 - 0.957344 2024-04-16T15:18:51.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1103 12 1832 1820 0.103114 0 - 0.957344 2024-04-16T15:18:52.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1104 12 1832 1820 0.103021 0 - 0.957344 2024-04-16T15:18:53.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1105 12 1832 1820 0.102927 0 - 0.957344 2024-04-16T15:18:54.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1106 12 1832 1820 0.102834 0 - 0.957344 2024-04-16T15:18:55.636 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1107 12 1832 1820 0.102742 0 - 0.957344 2024-04-16T15:18:56.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1108 12 1832 1820 0.102649 0 - 0.957344 2024-04-16T15:18:57.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1109 12 1832 1820 0.102556 0 - 0.957344 2024-04-16T15:18:58.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1110 12 1832 1820 0.102464 0 - 0.957344 2024-04-16T15:18:59.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1111 12 1832 1820 0.102372 0 - 0.957344 2024-04-16T15:19:00.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1112 12 1832 1820 0.10228 0 - 0.957344 2024-04-16T15:19:01.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1113 12 1832 1820 0.102188 0 - 0.957344 2024-04-16T15:19:02.637 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1114 12 1832 1820 0.102096 0 - 0.957344 2024-04-16T15:19:03.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1115 12 1832 1820 0.102004 0 - 0.957344 2024-04-16T15:19:04.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1116 12 1832 1820 0.101913 0 - 0.957344 2024-04-16T15:19:05.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1117 12 1832 1820 0.101822 0 - 0.957344 2024-04-16T15:19:06.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1118 12 1832 1820 0.101731 0 - 0.957344 2024-04-16T15:19:07.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1119 12 1832 1820 0.10164 0 - 0.957344 2024-04-16T15:19:08.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:19:08.641287+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:19:08.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:19:08.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1120 12 1832 1820 0.101549 0 - 0.957344 2024-04-16T15:19:09.638 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1121 12 1832 1820 0.101458 0 - 0.957344 2024-04-16T15:19:10.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1122 12 1832 1820 0.101368 0 - 0.957344 2024-04-16T15:19:11.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1123 12 1832 1820 0.101278 0 - 0.957344 2024-04-16T15:19:12.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1124 12 1832 1820 0.101188 0 - 0.957344 2024-04-16T15:19:13.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1125 12 1832 1820 0.101098 0 - 0.957344 2024-04-16T15:19:14.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1126 12 1832 1820 0.101008 0 - 0.957344 2024-04-16T15:19:15.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1127 12 1832 1820 0.100918 0 - 0.957344 2024-04-16T15:19:16.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1128 12 1832 1820 0.100829 0 - 0.957344 2024-04-16T15:19:17.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1129 12 1832 1820 0.100739 0 - 0.957344 2024-04-16T15:19:18.639 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1130 12 1832 1820 0.10065 0 - 0.957344 2024-04-16T15:19:19.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1131 12 1832 1820 0.100561 0 - 0.957344 2024-04-16T15:19:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:19:20.138 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:19:20.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1132 12 1832 1820 0.100472 0 - 0.957344 2024-04-16T15:19:21.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1133 12 1832 1820 0.100384 0 - 0.957344 2024-04-16T15:19:22.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:19:22.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:19:22.262+0000 7f3d3e8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:19:22.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1134 12 1832 1820 0.100295 0 - 0.957344 2024-04-16T15:19:23.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1135 12 1832 1820 0.100207 0 - 0.957344 2024-04-16T15:19:24.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1136 12 1832 1820 0.100119 0 - 0.957344 2024-04-16T15:19:25.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1137 12 1832 1820 0.100031 0 - 0.957344 2024-04-16T15:19:26.640 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1138 12 1832 1820 0.0999428 0 - 0.957344 2024-04-16T15:19:27.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1139 12 1832 1820 0.099855 0 - 0.957344 2024-04-16T15:19:28.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:19:28.643847+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:19:28.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:19:28.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1140 12 1832 1820 0.0997674 0 - 0.957344 2024-04-16T15:19:29.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1141 12 1832 1820 0.09968 0 - 0.957344 2024-04-16T15:19:30.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1142 12 1832 1820 0.0995927 0 - 0.957344 2024-04-16T15:19:31.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1143 12 1832 1820 0.0995056 0 - 0.957344 2024-04-16T15:19:32.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1144 12 1832 1820 0.0994186 0 - 0.957344 2024-04-16T15:19:33.641 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1145 12 1832 1820 0.0993318 0 - 0.957344 2024-04-16T15:19:34.642 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1146 12 1832 1820 0.0992451 0 - 0.957344 2024-04-16T15:19:35.642 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1147 12 1832 1820 0.0991585 0 - 0.957344 2024-04-16T15:19:36.642 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1148 12 1832 1820 0.0990722 0 - 0.957344 2024-04-16T15:19:37.642 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1149 12 1832 1820 0.0989859 0 - 0.957344 2024-04-16T15:19:38.642 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1150 12 1832 1820 0.0988999 0 - 0.957344 2024-04-16T15:19:39.642 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1151 12 1832 1820 0.0988139 0 - 0.957344 2024-04-16T15:19:40.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1152 12 1832 1820 0.0987282 0 - 0.957344 2024-04-16T15:19:41.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1153 12 1832 1820 0.0986425 0 - 0.957344 2024-04-16T15:19:42.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1154 12 1832 1820 0.0985571 0 - 0.957344 2024-04-16T15:19:43.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1155 12 1832 1820 0.0984717 0 - 0.957344 2024-04-16T15:19:44.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1156 12 1832 1820 0.0983865 0 - 0.957344 2024-04-16T15:19:45.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1157 12 1832 1820 0.0983015 0 - 0.957344 2024-04-16T15:19:46.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1158 12 1832 1820 0.0982166 0 - 0.957344 2024-04-16T15:19:47.643 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1159 12 1832 1820 0.0981319 0 - 0.957344 2024-04-16T15:19:48.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:19:48.646598+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:19:48.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:19:48.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1160 12 1832 1820 0.0980473 0 - 0.957344 2024-04-16T15:19:49.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1161 12 1832 1820 0.0979628 0 - 0.957344 2024-04-16T15:19:50.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1162 12 1832 1820 0.0978785 0 - 0.957344 2024-04-16T15:19:51.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1163 12 1832 1820 0.0977944 0 - 0.957344 2024-04-16T15:19:52.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1164 12 1832 1820 0.0977103 0 - 0.957344 2024-04-16T15:19:53.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1165 12 1832 1820 0.0976265 0 - 0.957344 2024-04-16T15:19:54.644 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1166 12 1832 1820 0.0975427 0 - 0.957344 2024-04-16T15:19:55.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1167 12 1832 1820 0.0974592 0 - 0.957344 2024-04-16T15:19:56.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1168 12 1832 1820 0.0973757 0 - 0.957344 2024-04-16T15:19:57.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1169 12 1832 1820 0.0972924 0 - 0.957344 2024-04-16T15:19:58.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1170 12 1832 1820 0.0972093 0 - 0.957344 2024-04-16T15:19:59.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1171 12 1832 1820 0.0971262 0 - 0.957344 2024-04-16T15:20:00.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1172 12 1832 1820 0.0970434 0 - 0.957344 2024-04-16T15:20:01.645 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1173 12 1832 1820 0.0969606 0 - 0.957344 2024-04-16T15:20:02.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1174 12 1832 1820 0.0968781 0 - 0.957344 2024-04-16T15:20:03.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1175 12 1832 1820 0.0967956 0 - 0.957344 2024-04-16T15:20:04.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1176 12 1832 1820 0.0967133 0 - 0.957344 2024-04-16T15:20:05.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1177 12 1832 1820 0.0966311 0 - 0.957344 2024-04-16T15:20:06.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1178 12 1832 1820 0.0965491 0 - 0.957344 2024-04-16T15:20:07.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1179 12 1832 1820 0.0964672 0 - 0.957344 2024-04-16T15:20:08.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:20:08.649395+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:20:08.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:20:08.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1180 12 1832 1820 0.0963855 0 - 0.957344 2024-04-16T15:20:09.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1181 12 1832 1820 0.0963038 0 - 0.957344 2024-04-16T15:20:10.646 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1182 12 1832 1820 0.0962224 0 - 0.957344 2024-04-16T15:20:11.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1183 12 1832 1820 0.096141 0 - 0.957344 2024-04-16T15:20:12.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1184 12 1832 1820 0.0960598 0 - 0.957344 2024-04-16T15:20:13.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1185 12 1832 1820 0.0959788 0 - 0.957344 2024-04-16T15:20:14.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1186 12 1832 1820 0.0958978 0 - 0.957344 2024-04-16T15:20:15.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1187 12 1832 1820 0.095817 0 - 0.957344 2024-04-16T15:20:16.647 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1188 12 1832 1820 0.0957364 0 - 0.957344 2024-04-16T15:20:17.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1189 12 1832 1820 0.0956559 0 - 0.957344 2024-04-16T15:20:18.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1190 12 1832 1820 0.0955755 0 - 0.957344 2024-04-16T15:20:19.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1191 12 1832 1820 0.0954952 0 - 0.957344 2024-04-16T15:20:20.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1192 12 1832 1820 0.0954151 0 - 0.957344 2024-04-16T15:20:21.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1193 12 1832 1820 0.0953351 0 - 0.957344 2024-04-16T15:20:22.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1194 12 1832 1820 0.0952553 0 - 0.957344 2024-04-16T15:20:23.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1195 12 1832 1820 0.0951756 0 - 0.957344 2024-04-16T15:20:24.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1196 12 1832 1820 0.095096 0 - 0.957344 2024-04-16T15:20:25.648 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1197 12 1832 1820 0.0950166 0 - 0.957344 2024-04-16T15:20:26.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1198 12 1832 1820 0.0949373 0 - 0.957344 2024-04-16T15:20:27.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1199 12 1832 1820 0.0948581 0 - 0.957344 2024-04-16T15:20:28.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:20:28.652148+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:20:28.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:20:28.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1200 12 1832 1820 0.094779 0 - 0.957344 2024-04-16T15:20:29.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1201 12 1832 1820 0.0947001 0 - 0.957344 2024-04-16T15:20:30.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1202 12 1832 1820 0.0946213 0 - 0.957344 2024-04-16T15:20:31.649 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1203 12 1832 1820 0.0945427 0 - 0.957344 2024-04-16T15:20:32.650 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1204 12 1832 1820 0.0944641 0 - 0.957344 2024-04-16T15:20:33.650 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1205 12 1832 1820 0.0943857 0 - 0.957344 2024-04-16T15:20:34.650 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1206 12 1832 1820 0.0943075 0 - 0.957344 2024-04-16T15:20:35.650 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1207 12 1832 1820 0.0942294 0 - 0.957344 2024-04-16T15:20:36.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1208 12 1832 1820 0.0941513 0 - 0.957344 2024-04-16T15:20:37.650 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1209 12 1832 1820 0.0940735 0 - 0.957344 2024-04-16T15:20:38.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1210 12 1832 1820 0.0939957 0 - 0.957344 2024-04-16T15:20:39.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1211 12 1832 1820 0.0939181 0 - 0.957344 2024-04-16T15:20:40.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1212 12 1832 1820 0.0938406 0 - 0.957344 2024-04-16T15:20:41.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1213 12 1832 1820 0.0937632 0 - 0.957344 2024-04-16T15:20:42.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1214 12 1832 1820 0.093686 0 - 0.957344 2024-04-16T15:20:43.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1215 12 1832 1820 0.0936089 0 - 0.957344 2024-04-16T15:20:44.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1216 12 1832 1820 0.0935319 0 - 0.957344 2024-04-16T15:20:45.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1217 12 1832 1820 0.0934551 0 - 0.957344 2024-04-16T15:20:46.651 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1218 12 1832 1820 0.0933783 0 - 0.957344 2024-04-16T15:20:47.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1219 12 1832 1820 0.0933017 0 - 0.957344 2024-04-16T15:20:48.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:20:48.654902+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:20:48.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:20:48.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1220 12 1832 1820 0.0932253 0 - 0.957344 2024-04-16T15:20:49.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1221 12 1832 1820 0.0931489 0 - 0.957344 2024-04-16T15:20:50.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1222 12 1832 1820 0.0930727 0 - 0.957344 2024-04-16T15:20:51.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1223 12 1832 1820 0.0929966 0 - 0.957344 2024-04-16T15:20:52.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1224 12 1832 1820 0.0929206 0 - 0.957344 2024-04-16T15:20:53.652 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1225 12 1832 1820 0.0928448 0 - 0.957344 2024-04-16T15:20:54.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1226 12 1832 1820 0.092769 0 - 0.957344 2024-04-16T15:20:55.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1227 12 1832 1820 0.0926934 0 - 0.957344 2024-04-16T15:20:56.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1228 12 1832 1820 0.0926179 0 - 0.957344 2024-04-16T15:20:57.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1229 12 1832 1820 0.0925426 0 - 0.957344 2024-04-16T15:20:58.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1230 12 1832 1820 0.0924673 0 - 0.957344 2024-04-16T15:20:59.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1231 12 1832 1820 0.0923922 0 - 0.957344 2024-04-16T15:21:00.653 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1232 12 1832 1820 0.0923172 0 - 0.957344 2024-04-16T15:21:01.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1233 12 1832 1820 0.0922424 0 - 0.957344 2024-04-16T15:21:02.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1234 12 1832 1820 0.0921676 0 - 0.957344 2024-04-16T15:21:03.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1235 12 1832 1820 0.092093 0 - 0.957344 2024-04-16T15:21:04.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1236 12 1832 1820 0.0920185 0 - 0.957344 2024-04-16T15:21:05.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1237 12 1832 1820 0.0919441 0 - 0.957344 2024-04-16T15:21:06.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1238 12 1832 1820 0.0918698 0 - 0.957344 2024-04-16T15:21:07.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1239 12 1832 1820 0.0917957 0 - 0.957344 2024-04-16T15:21:08.654 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:21:08.657508+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:21:08.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:21:08.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1240 12 1832 1820 0.0917216 0 - 0.957344 2024-04-16T15:21:09.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1241 12 1832 1820 0.0916477 0 - 0.957344 2024-04-16T15:21:10.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1242 12 1832 1820 0.0915739 0 - 0.957344 2024-04-16T15:21:11.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1243 12 1832 1820 0.0915003 0 - 0.957344 2024-04-16T15:21:12.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1244 12 1832 1820 0.0914267 0 - 0.957344 2024-04-16T15:21:13.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1245 12 1832 1820 0.0913533 0 - 0.957344 2024-04-16T15:21:14.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1246 12 1832 1820 0.09128 0 - 0.957344 2024-04-16T15:21:15.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1247 12 1832 1820 0.0912068 0 - 0.957344 2024-04-16T15:21:16.655 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1248 12 1832 1820 0.0911337 0 - 0.957344 2024-04-16T15:21:17.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1249 12 1832 1820 0.0910607 0 - 0.957344 2024-04-16T15:21:18.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1250 12 1832 1820 0.0909879 0 - 0.957344 2024-04-16T15:21:19.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1251 12 1832 1820 0.0909151 0 - 0.957344 2024-04-16T15:21:20.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1252 12 1832 1820 0.0908425 0 - 0.957344 2024-04-16T15:21:21.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1253 12 1832 1820 0.09077 0 - 0.957344 2024-04-16T15:21:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:21:22.180 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:21:22.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1254 12 1832 1820 0.0906976 0 - 0.957344 2024-04-16T15:21:23.656 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1255 12 1832 1820 0.0906254 0 - 0.957344 2024-04-16T15:21:24.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:21:24.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:21:24.305+0000 7f778919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:21:24.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1256 12 1832 1820 0.0905532 0 - 0.957344 2024-04-16T15:21:25.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1257 12 1832 1820 0.0904812 0 - 0.957344 2024-04-16T15:21:26.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1258 12 1832 1820 0.0904092 0 - 0.957344 2024-04-16T15:21:27.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1259 12 1832 1820 0.0903374 0 - 0.957344 2024-04-16T15:21:28.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:21:28.660267+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:21:28.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:21:28.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1260 12 1832 1820 0.0902657 0 - 0.957344 2024-04-16T15:21:29.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1261 12 1832 1820 0.0901941 0 - 0.957344 2024-04-16T15:21:30.657 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1262 12 1832 1820 0.0901227 0 - 0.957344 2024-04-16T15:21:31.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1263 12 1832 1820 0.0900513 0 - 0.957344 2024-04-16T15:21:32.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1264 12 1832 1820 0.0899801 0 - 0.957344 2024-04-16T15:21:33.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1265 12 1832 1820 0.0899089 0 - 0.957344 2024-04-16T15:21:34.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1266 12 1832 1820 0.0898379 0 - 0.957344 2024-04-16T15:21:35.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1267 12 1832 1820 0.089767 0 - 0.957344 2024-04-16T15:21:36.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1268 12 1832 1820 0.0896962 0 - 0.957344 2024-04-16T15:21:37.658 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1269 12 1832 1820 0.0896255 0 - 0.957344 2024-04-16T15:21:38.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1270 12 1832 1820 0.089555 0 - 0.957344 2024-04-16T15:21:39.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1271 12 1832 1820 0.0894845 0 - 0.957344 2024-04-16T15:21:40.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1272 12 1832 1820 0.0894142 0 - 0.957344 2024-04-16T15:21:41.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1273 12 1832 1820 0.0893439 0 - 0.957344 2024-04-16T15:21:42.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1274 12 1832 1820 0.0892738 0 - 0.957344 2024-04-16T15:21:43.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1275 12 1832 1820 0.0892038 0 - 0.957344 2024-04-16T15:21:44.659 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1276 12 1832 1820 0.0891339 0 - 0.957344 2024-04-16T15:21:45.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1277 12 1832 1820 0.0890641 0 - 0.957344 2024-04-16T15:21:46.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1278 12 1832 1820 0.0889944 0 - 0.957344 2024-04-16T15:21:47.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1279 12 1832 1820 0.0889248 0 - 0.957344 2024-04-16T15:21:48.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:21:48.662996+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:21:48.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:21:48.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1280 12 1832 1820 0.0888553 0 - 0.957344 2024-04-16T15:21:49.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1281 12 1832 1820 0.088786 0 - 0.957344 2024-04-16T15:21:50.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1282 12 1832 1820 0.0887167 0 - 0.957344 2024-04-16T15:21:51.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1283 12 1832 1820 0.0886476 0 - 0.957344 2024-04-16T15:21:52.660 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1284 12 1832 1820 0.0885785 0 - 0.957344 2024-04-16T15:21:53.661 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1285 12 1832 1820 0.0885096 0 - 0.957344 2024-04-16T15:21:54.661 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1286 12 1832 1820 0.0884408 0 - 0.957344 2024-04-16T15:21:55.661 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1287 12 1832 1820 0.088372 0 - 0.957344 2024-04-16T15:21:56.661 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1288 12 1832 1820 0.0883034 0 - 0.957344 2024-04-16T15:21:57.661 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1289 12 1832 1820 0.0882349 0 - 0.957344 2024-04-16T15:21:58.661 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1290 12 1832 1820 0.0881665 0 - 0.957344 2024-04-16T15:21:59.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1291 12 1832 1820 0.0880982 0 - 0.957344 2024-04-16T15:22:00.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1292 12 1832 1820 0.08803 0 - 0.957344 2024-04-16T15:22:01.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1293 12 1832 1820 0.087962 0 - 0.957344 2024-04-16T15:22:02.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1294 12 1832 1820 0.087894 0 - 0.957344 2024-04-16T15:22:03.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1295 12 1832 1820 0.0878261 0 - 0.957344 2024-04-16T15:22:04.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1296 12 1832 1820 0.0877583 0 - 0.957344 2024-04-16T15:22:05.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1297 12 1832 1820 0.0876907 0 - 0.957344 2024-04-16T15:22:06.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1298 12 1832 1820 0.0876231 0 - 0.957344 2024-04-16T15:22:07.662 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1299 12 1832 1820 0.0875557 0 - 0.957344 2024-04-16T15:22:08.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:22:08.665712+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:22:08.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:22:08.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1300 12 1832 1820 0.0874883 0 - 0.957344 2024-04-16T15:22:09.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1301 12 1832 1820 0.0874211 0 - 0.957344 2024-04-16T15:22:10.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1302 12 1832 1820 0.0873539 0 - 0.957344 2024-04-16T15:22:11.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1303 12 1832 1820 0.0872869 0 - 0.957344 2024-04-16T15:22:12.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1304 12 1832 1820 0.0872199 0 - 0.957344 2024-04-16T15:22:13.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1305 12 1832 1820 0.0871531 0 - 0.957344 2024-04-16T15:22:14.663 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1306 12 1832 1820 0.0870864 0 - 0.957344 2024-04-16T15:22:15.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1307 12 1832 1820 0.0870197 0 - 0.957344 2024-04-16T15:22:16.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1308 12 1832 1820 0.0869532 0 - 0.957344 2024-04-16T15:22:17.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1309 12 1832 1820 0.0868868 0 - 0.957344 2024-04-16T15:22:18.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1310 12 1832 1820 0.0868205 0 - 0.957344 2024-04-16T15:22:19.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1311 12 1832 1820 0.0867542 0 - 0.957344 2024-04-16T15:22:20.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1312 12 1832 1820 0.0866881 0 - 0.957344 2024-04-16T15:22:21.664 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1313 12 1832 1820 0.0866221 0 - 0.957344 2024-04-16T15:22:22.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1314 12 1832 1820 0.0865562 0 - 0.957344 2024-04-16T15:22:23.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1315 12 1832 1820 0.0864904 0 - 0.957344 2024-04-16T15:22:24.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1316 12 1832 1820 0.0864246 0 - 0.957344 2024-04-16T15:22:25.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1317 12 1832 1820 0.086359 0 - 0.957344 2024-04-16T15:22:26.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1318 12 1832 1820 0.0862935 0 - 0.957344 2024-04-16T15:22:27.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1319 12 1832 1820 0.0862281 0 - 0.957344 2024-04-16T15:22:28.665 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:22:28.668490+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:22:28.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:22:28.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1320 12 1832 1820 0.0861627 0 - 0.957344 2024-04-16T15:22:29.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1321 12 1832 1820 0.0860975 0 - 0.957344 2024-04-16T15:22:30.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1322 12 1832 1820 0.0860324 0 - 0.957344 2024-04-16T15:22:31.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1323 12 1832 1820 0.0859673 0 - 0.957344 2024-04-16T15:22:32.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1324 12 1832 1820 0.0859024 0 - 0.957344 2024-04-16T15:22:33.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1325 12 1832 1820 0.0858376 0 - 0.957344 2024-04-16T15:22:34.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1326 12 1832 1820 0.0857729 0 - 0.957344 2024-04-16T15:22:35.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1327 12 1832 1820 0.0857082 0 - 0.957344 2024-04-16T15:22:36.666 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1328 12 1832 1820 0.0856437 0 - 0.957344 2024-04-16T15:22:37.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1329 12 1832 1820 0.0855792 0 - 0.957344 2024-04-16T15:22:38.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1330 12 1832 1820 0.0855149 0 - 0.957344 2024-04-16T15:22:39.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1331 12 1832 1820 0.0854506 0 - 0.957344 2024-04-16T15:22:40.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1332 12 1832 1820 0.0853865 0 - 0.957344 2024-04-16T15:22:41.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1333 12 1832 1820 0.0853224 0 - 0.957344 2024-04-16T15:22:42.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1334 12 1832 1820 0.0852585 0 - 0.957344 2024-04-16T15:22:43.667 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1335 12 1832 1820 0.0851946 0 - 0.957344 2024-04-16T15:22:44.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1336 12 1832 1820 0.0851308 0 - 0.957344 2024-04-16T15:22:45.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1337 12 1832 1820 0.0850672 0 - 0.957344 2024-04-16T15:22:46.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1338 12 1832 1820 0.0850036 0 - 0.957344 2024-04-16T15:22:47.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1339 12 1832 1820 0.0849401 0 - 0.957344 2024-04-16T15:22:48.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:22:48.671188+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:22:48.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:22:48.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1340 12 1832 1820 0.0848767 0 - 0.957344 2024-04-16T15:22:49.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1341 12 1832 1820 0.0848134 0 - 0.957344 2024-04-16T15:22:50.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1342 12 1832 1820 0.0847502 0 - 0.957344 2024-04-16T15:22:51.668 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1343 12 1832 1820 0.0846871 0 - 0.957344 2024-04-16T15:22:52.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1344 12 1832 1820 0.0846241 0 - 0.957344 2024-04-16T15:22:53.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1345 12 1832 1820 0.0845612 0 - 0.957344 2024-04-16T15:22:54.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1346 12 1832 1820 0.0844984 0 - 0.957344 2024-04-16T15:22:55.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1347 12 1832 1820 0.0844356 0 - 0.957344 2024-04-16T15:22:56.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1348 12 1832 1820 0.084373 0 - 0.957344 2024-04-16T15:22:57.669 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1349 12 1832 1820 0.0843105 0 - 0.957344 2024-04-16T15:22:58.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1350 12 1832 1820 0.084248 0 - 0.957344 2024-04-16T15:22:59.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1351 12 1832 1820 0.0841856 0 - 0.957344 2024-04-16T15:23:00.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1352 12 1832 1820 0.0841234 0 - 0.957344 2024-04-16T15:23:01.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1353 12 1832 1820 0.0840612 0 - 0.957344 2024-04-16T15:23:02.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1354 12 1832 1820 0.0839991 0 - 0.957344 2024-04-16T15:23:03.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1355 12 1832 1820 0.0839371 0 - 0.957344 2024-04-16T15:23:04.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1356 12 1832 1820 0.0838752 0 - 0.957344 2024-04-16T15:23:05.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1357 12 1832 1820 0.0838134 0 - 0.957344 2024-04-16T15:23:06.670 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1358 12 1832 1820 0.0837517 0 - 0.957344 2024-04-16T15:23:07.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1359 12 1832 1820 0.0836901 0 - 0.957344 2024-04-16T15:23:08.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:23:08.673851+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:23:08.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:23:08.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1360 12 1832 1820 0.0836285 0 - 0.957344 2024-04-16T15:23:09.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1361 12 1832 1820 0.0835671 0 - 0.957344 2024-04-16T15:23:10.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1362 12 1832 1820 0.0835057 0 - 0.957344 2024-04-16T15:23:11.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1363 12 1832 1820 0.0834445 0 - 0.957344 2024-04-16T15:23:12.671 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1364 12 1832 1820 0.0833833 0 - 0.957344 2024-04-16T15:23:13.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1365 12 1832 1820 0.0833222 0 - 0.957344 2024-04-16T15:23:14.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1366 12 1832 1820 0.0832612 0 - 0.957344 2024-04-16T15:23:15.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1367 12 1832 1820 0.0832003 0 - 0.957344 2024-04-16T15:23:16.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1368 12 1832 1820 0.0831395 0 - 0.957344 2024-04-16T15:23:17.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1369 12 1832 1820 0.0830787 0 - 0.957344 2024-04-16T15:23:18.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1370 12 1832 1820 0.0830181 0 - 0.957344 2024-04-16T15:23:19.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1371 12 1832 1820 0.0829575 0 - 0.957344 2024-04-16T15:23:20.672 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1372 12 1832 1820 0.0828971 0 - 0.957344 2024-04-16T15:23:21.673 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1373 12 1832 1820 0.0828367 0 - 0.957344 2024-04-16T15:23:22.673 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1374 12 1832 1820 0.0827764 0 - 0.957344 2024-04-16T15:23:23.673 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1375 12 1832 1820 0.0827162 0 - 0.957344 2024-04-16T15:23:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:23:24.222 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:23:24.673 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1376 12 1832 1820 0.0826561 0 - 0.957344 2024-04-16T15:23:25.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1377 12 1832 1820 0.0825961 0 - 0.957344 2024-04-16T15:23:26.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:23:26.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:23:26.347+0000 7f113d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:23:26.673 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1378 12 1832 1820 0.0825361 0 - 0.957344 2024-04-16T15:23:27.673 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1379 12 1832 1820 0.0824763 0 - 0.957344 2024-04-16T15:23:28.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:23:28.676523+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:23:28.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:23:28.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1380 12 1832 1820 0.0824165 0 - 0.957344 2024-04-16T15:23:29.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1381 12 1832 1820 0.0823568 0 - 0.957344 2024-04-16T15:23:30.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1382 12 1832 1820 0.0822972 0 - 0.957344 2024-04-16T15:23:31.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1383 12 1832 1820 0.0822377 0 - 0.957344 2024-04-16T15:23:32.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1384 12 1832 1820 0.0821783 0 - 0.957344 2024-04-16T15:23:33.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1385 12 1832 1820 0.082119 0 - 0.957344 2024-04-16T15:23:34.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1386 12 1832 1820 0.0820597 0 - 0.957344 2024-04-16T15:23:35.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1387 12 1832 1820 0.0820006 0 - 0.957344 2024-04-16T15:23:36.674 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1388 12 1832 1820 0.0819415 0 - 0.957344 2024-04-16T15:23:37.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1389 12 1832 1820 0.0818825 0 - 0.957344 2024-04-16T15:23:38.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1390 12 1832 1820 0.0818236 0 - 0.957344 2024-04-16T15:23:39.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1391 12 1832 1820 0.0817648 0 - 0.957344 2024-04-16T15:23:40.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1392 12 1832 1820 0.081706 0 - 0.957344 2024-04-16T15:23:41.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1393 12 1832 1820 0.0816474 0 - 0.957344 2024-04-16T15:23:42.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1394 12 1832 1820 0.0815888 0 - 0.957344 2024-04-16T15:23:43.675 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1395 12 1832 1820 0.0815303 0 - 0.957344 2024-04-16T15:23:44.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1396 12 1832 1820 0.0814719 0 - 0.957344 2024-04-16T15:23:45.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1397 12 1832 1820 0.0814136 0 - 0.957344 2024-04-16T15:23:46.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1398 12 1832 1820 0.0813554 0 - 0.957344 2024-04-16T15:23:47.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1399 12 1832 1820 0.0812972 0 - 0.957344 2024-04-16T15:23:48.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:23:48.679142+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:23:48.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:23:48.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1400 12 1832 1820 0.0812391 0 - 0.957344 2024-04-16T15:23:49.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1401 12 1832 1820 0.0811812 0 - 0.957344 2024-04-16T15:23:50.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1402 12 1832 1820 0.0811233 0 - 0.957344 2024-04-16T15:23:51.676 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1403 12 1832 1820 0.0810654 0 - 0.957344 2024-04-16T15:23:52.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1404 12 1832 1820 0.0810077 0 - 0.957344 2024-04-16T15:23:53.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1405 12 1832 1820 0.08095 0 - 0.957344 2024-04-16T15:23:54.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1406 12 1832 1820 0.0808925 0 - 0.957344 2024-04-16T15:23:55.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1407 12 1832 1820 0.080835 0 - 0.957344 2024-04-16T15:23:56.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1408 12 1832 1820 0.0807776 0 - 0.957344 2024-04-16T15:23:57.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1409 12 1832 1820 0.0807202 0 - 0.957344 2024-04-16T15:23:58.677 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1410 12 1832 1820 0.080663 0 - 0.957344 2024-04-16T15:23:59.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1411 12 1832 1820 0.0806058 0 - 0.957344 2024-04-16T15:24:00.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1412 12 1832 1820 0.0805487 0 - 0.957344 2024-04-16T15:24:01.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1413 12 1832 1820 0.0804917 0 - 0.957344 2024-04-16T15:24:02.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1414 12 1832 1820 0.0804348 0 - 0.957344 2024-04-16T15:24:03.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1415 12 1832 1820 0.080378 0 - 0.957344 2024-04-16T15:24:04.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1416 12 1832 1820 0.0803212 0 - 0.957344 2024-04-16T15:24:05.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1417 12 1832 1820 0.0802645 0 - 0.957344 2024-04-16T15:24:06.678 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1418 12 1832 1820 0.0802079 0 - 0.957344 2024-04-16T15:24:07.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1419 12 1832 1820 0.0801514 0 - 0.957344 2024-04-16T15:24:08.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:24:08.681758+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:24:08.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:24:08.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1420 12 1832 1820 0.0800949 0 - 0.957344 2024-04-16T15:24:09.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1421 12 1832 1820 0.0800386 0 - 0.957344 2024-04-16T15:24:10.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1422 12 1832 1820 0.0799823 0 - 0.957344 2024-04-16T15:24:11.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1423 12 1832 1820 0.0799261 0 - 0.957344 2024-04-16T15:24:12.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1424 12 1832 1820 0.0798699 0 - 0.957344 2024-04-16T15:24:13.679 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1425 12 1832 1820 0.0798139 0 - 0.957344 2024-04-16T15:24:14.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1426 12 1832 1820 0.0797579 0 - 0.957344 2024-04-16T15:24:15.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1427 12 1832 1820 0.079702 0 - 0.957344 2024-04-16T15:24:16.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1428 12 1832 1820 0.0796462 0 - 0.957344 2024-04-16T15:24:17.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1429 12 1832 1820 0.0795905 0 - 0.957344 2024-04-16T15:24:18.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1430 12 1832 1820 0.0795348 0 - 0.957344 2024-04-16T15:24:19.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1431 12 1832 1820 0.0794792 0 - 0.957344 2024-04-16T15:24:20.680 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1432 12 1832 1820 0.0794237 0 - 0.957344 2024-04-16T15:24:21.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1433 12 1832 1820 0.0793683 0 - 0.957344 2024-04-16T15:24:22.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1434 12 1832 1820 0.079313 0 - 0.957344 2024-04-16T15:24:23.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1435 12 1832 1820 0.0792577 0 - 0.957344 2024-04-16T15:24:24.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1436 12 1832 1820 0.0792025 0 - 0.957344 2024-04-16T15:24:25.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1437 12 1832 1820 0.0791474 0 - 0.957344 2024-04-16T15:24:26.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1438 12 1832 1820 0.0790924 0 - 0.957344 2024-04-16T15:24:27.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1439 12 1832 1820 0.0790374 0 - 0.957344 2024-04-16T15:24:28.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:24:28.684464+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:24:28.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:24:28.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1440 12 1832 1820 0.0789825 0 - 0.957344 2024-04-16T15:24:29.681 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1441 12 1832 1820 0.0789277 0 - 0.957344 2024-04-16T15:24:30.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1442 12 1832 1820 0.078873 0 - 0.957344 2024-04-16T15:24:31.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1443 12 1832 1820 0.0788183 0 - 0.957344 2024-04-16T15:24:32.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1444 12 1832 1820 0.0787637 0 - 0.957344 2024-04-16T15:24:33.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1445 12 1832 1820 0.0787092 0 - 0.957344 2024-04-16T15:24:34.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1446 12 1832 1820 0.0786548 0 - 0.957344 2024-04-16T15:24:35.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1447 12 1832 1820 0.0786004 0 - 0.957344 2024-04-16T15:24:36.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1448 12 1832 1820 0.0785461 0 - 0.957344 2024-04-16T15:24:37.682 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1449 12 1832 1820 0.0784919 0 - 0.957344 2024-04-16T15:24:38.683 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1450 12 1832 1820 0.0784378 0 - 0.957344 2024-04-16T15:24:39.683 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1451 12 1832 1820 0.0783837 0 - 0.957344 2024-04-16T15:24:40.683 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1452 12 1832 1820 0.0783298 0 - 0.957344 2024-04-16T15:24:41.683 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1453 12 1832 1820 0.0782758 0 - 0.957344 2024-04-16T15:24:42.683 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1454 12 1832 1820 0.078222 0 - 0.957344 2024-04-16T15:24:43.683 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1455 12 1832 1820 0.0781683 0 - 0.957344 2024-04-16T15:24:44.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1456 12 1832 1820 0.0781146 0 - 0.957344 2024-04-16T15:24:45.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1457 12 1832 1820 0.078061 0 - 0.957344 2024-04-16T15:24:46.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1458 12 1832 1820 0.0780074 0 - 0.957344 2024-04-16T15:24:47.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1459 12 1832 1820 0.0779539 0 - 0.957344 2024-04-16T15:24:48.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:24:48.687132+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:24:48.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:24:48.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1460 12 1832 1820 0.0779006 0 - 0.957344 2024-04-16T15:24:49.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1461 12 1832 1820 0.0778472 0 - 0.957344 2024-04-16T15:24:50.684 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1462 12 1832 1820 0.077794 0 - 0.957344 2024-04-16T15:24:51.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1463 12 1832 1820 0.0777408 0 - 0.957344 2024-04-16T15:24:52.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1464 12 1832 1820 0.0776877 0 - 0.957344 2024-04-16T15:24:53.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1465 12 1832 1820 0.0776347 0 - 0.957344 2024-04-16T15:24:54.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1466 12 1832 1820 0.0775817 0 - 0.957344 2024-04-16T15:24:55.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1467 12 1832 1820 0.0775288 0 - 0.957344 2024-04-16T15:24:56.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1468 12 1832 1820 0.077476 0 - 0.957344 2024-04-16T15:24:57.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1469 12 1832 1820 0.0774233 0 - 0.957344 2024-04-16T15:24:58.685 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1470 12 1832 1820 0.0773706 0 - 0.957344 2024-04-16T15:24:59.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1471 12 1832 1820 0.077318 0 - 0.957344 2024-04-16T15:25:00.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1472 12 1832 1820 0.0772655 0 - 0.957344 2024-04-16T15:25:01.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1473 12 1832 1820 0.077213 0 - 0.957344 2024-04-16T15:25:02.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1474 12 1832 1820 0.0771607 0 - 0.957344 2024-04-16T15:25:03.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1475 12 1832 1820 0.0771083 0 - 0.957344 2024-04-16T15:25:04.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1476 12 1832 1820 0.0770561 0 - 0.957344 2024-04-16T15:25:05.686 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1477 12 1832 1820 0.0770039 0 - 0.957344 2024-04-16T15:25:06.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1478 12 1832 1820 0.0769518 0 - 0.957344 2024-04-16T15:25:07.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1479 12 1832 1820 0.0768998 0 - 0.957344 2024-04-16T15:25:08.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:25:08.689924+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:25:08.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:25:08.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1480 12 1832 1820 0.0768478 0 - 0.957344 2024-04-16T15:25:09.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1481 12 1832 1820 0.0767959 0 - 0.957344 2024-04-16T15:25:10.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1482 12 1832 1820 0.0767441 0 - 0.957344 2024-04-16T15:25:11.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1483 12 1832 1820 0.0766924 0 - 0.957344 2024-04-16T15:25:12.687 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1484 12 1832 1820 0.0766407 0 - 0.957344 2024-04-16T15:25:13.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1485 12 1832 1820 0.0765891 0 - 0.957344 2024-04-16T15:25:14.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1486 12 1832 1820 0.0765375 0 - 0.957344 2024-04-16T15:25:15.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1487 12 1832 1820 0.0764861 0 - 0.957344 2024-04-16T15:25:16.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1488 12 1832 1820 0.0764347 0 - 0.957344 2024-04-16T15:25:17.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1489 12 1832 1820 0.0763833 0 - 0.957344 2024-04-16T15:25:18.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1490 12 1832 1820 0.0763321 0 - 0.957344 2024-04-16T15:25:19.688 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1491 12 1832 1820 0.0762809 0 - 0.957344 2024-04-16T15:25:20.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1492 12 1832 1820 0.0762298 0 - 0.957344 2024-04-16T15:25:21.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1493 12 1832 1820 0.0761787 0 - 0.957344 2024-04-16T15:25:22.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1494 12 1832 1820 0.0761277 0 - 0.957344 2024-04-16T15:25:23.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1495 12 1832 1820 0.0760768 0 - 0.957344 2024-04-16T15:25:24.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1496 12 1832 1820 0.0760259 0 - 0.957344 2024-04-16T15:25:25.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1497 12 1832 1820 0.0759751 0 - 0.957344 2024-04-16T15:25:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:25:26.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:25:26.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1498 12 1832 1820 0.0759244 0 - 0.957344 2024-04-16T15:25:27.689 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1499 12 1832 1820 0.0758738 0 - 0.957344 2024-04-16T15:25:28.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:25:28.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:25:28.388+0000 7fd13299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:25:28.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:25:28.692633+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:25:28.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:25:28.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1500 12 1832 1820 0.0758232 0 - 0.957344 2024-04-16T15:25:29.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1501 12 1832 1820 0.0757727 0 - 0.957344 2024-04-16T15:25:30.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1502 12 1832 1820 0.0757222 0 - 0.957344 2024-04-16T15:25:31.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1503 12 1832 1820 0.0756719 0 - 0.957344 2024-04-16T15:25:32.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1504 12 1832 1820 0.0756215 0 - 0.957344 2024-04-16T15:25:33.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1505 12 1832 1820 0.0755713 0 - 0.957344 2024-04-16T15:25:34.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1506 12 1832 1820 0.0755211 0 - 0.957344 2024-04-16T15:25:35.690 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1507 12 1832 1820 0.075471 0 - 0.957344 2024-04-16T15:25:36.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1508 12 1832 1820 0.075421 0 - 0.957344 2024-04-16T15:25:37.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1509 12 1832 1820 0.075371 0 - 0.957344 2024-04-16T15:25:38.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1510 12 1832 1820 0.0753211 0 - 0.957344 2024-04-16T15:25:39.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1511 12 1832 1820 0.0752712 0 - 0.957344 2024-04-16T15:25:40.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1512 12 1832 1820 0.0752214 0 - 0.957344 2024-04-16T15:25:41.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1513 12 1832 1820 0.0751717 0 - 0.957344 2024-04-16T15:25:42.691 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1514 12 1832 1820 0.0751221 0 - 0.957344 2024-04-16T15:25:43.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1515 12 1832 1820 0.0750725 0 - 0.957344 2024-04-16T15:25:44.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1516 12 1832 1820 0.075023 0 - 0.957344 2024-04-16T15:25:45.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1517 12 1832 1820 0.0749735 0 - 0.957344 2024-04-16T15:25:46.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1518 12 1832 1820 0.0749241 0 - 0.957344 2024-04-16T15:25:47.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1519 12 1832 1820 0.0748748 0 - 0.957344 2024-04-16T15:25:48.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:25:48.695230+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:25:48.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:25:48.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1520 12 1832 1820 0.0748255 0 - 0.957344 2024-04-16T15:25:49.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1521 12 1832 1820 0.0747763 0 - 0.957344 2024-04-16T15:25:50.692 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1522 12 1832 1820 0.0747272 0 - 0.957344 2024-04-16T15:25:51.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1523 12 1832 1820 0.0746781 0 - 0.957344 2024-04-16T15:25:52.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1524 12 1832 1820 0.0746291 0 - 0.957344 2024-04-16T15:25:53.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1525 12 1832 1820 0.0745802 0 - 0.957344 2024-04-16T15:25:54.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1526 12 1832 1820 0.0745313 0 - 0.957344 2024-04-16T15:25:55.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1527 12 1832 1820 0.0744825 0 - 0.957344 2024-04-16T15:25:56.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1528 12 1832 1820 0.0744338 0 - 0.957344 2024-04-16T15:25:57.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1529 12 1832 1820 0.0743851 0 - 0.957344 2024-04-16T15:25:58.693 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1530 12 1832 1820 0.0743365 0 - 0.957344 2024-04-16T15:25:59.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1531 12 1832 1820 0.0742879 0 - 0.957344 2024-04-16T15:26:00.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1532 12 1832 1820 0.0742394 0 - 0.957344 2024-04-16T15:26:01.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1533 12 1832 1820 0.074191 0 - 0.957344 2024-04-16T15:26:02.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1534 12 1832 1820 0.0741426 0 - 0.957344 2024-04-16T15:26:03.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1535 12 1832 1820 0.0740943 0 - 0.957344 2024-04-16T15:26:04.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1536 12 1832 1820 0.0740461 0 - 0.957344 2024-04-16T15:26:05.694 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1537 12 1832 1820 0.0739979 0 - 0.957344 2024-04-16T15:26:06.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1538 12 1832 1820 0.0739498 0 - 0.957344 2024-04-16T15:26:07.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1539 12 1832 1820 0.0739018 0 - 0.957344 2024-04-16T15:26:08.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:26:08.697993+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:26:08.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:26:08.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1540 12 1832 1820 0.0738538 0 - 0.957344 2024-04-16T15:26:09.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1541 12 1832 1820 0.0738058 0 - 0.957344 2024-04-16T15:26:10.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1542 12 1832 1820 0.073758 0 - 0.957344 2024-04-16T15:26:11.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1543 12 1832 1820 0.0737102 0 - 0.957344 2024-04-16T15:26:12.695 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1544 12 1832 1820 0.0736624 0 - 0.957344 2024-04-16T15:26:13.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1545 12 1832 1820 0.0736148 0 - 0.957344 2024-04-16T15:26:14.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1546 12 1832 1820 0.0735671 0 - 0.957344 2024-04-16T15:26:15.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1547 12 1832 1820 0.0735196 0 - 0.957344 2024-04-16T15:26:16.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1548 12 1832 1820 0.0734721 0 - 0.957344 2024-04-16T15:26:17.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1549 12 1832 1820 0.0734247 0 - 0.957344 2024-04-16T15:26:18.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1550 12 1832 1820 0.0733773 0 - 0.957344 2024-04-16T15:26:19.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1551 12 1832 1820 0.07333 0 - 0.957344 2024-04-16T15:26:20.696 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1552 12 1832 1820 0.0732827 0 - 0.957344 2024-04-16T15:26:21.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1553 12 1832 1820 0.0732355 0 - 0.957344 2024-04-16T15:26:22.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1554 12 1832 1820 0.0731884 0 - 0.957344 2024-04-16T15:26:23.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1555 12 1832 1820 0.0731413 0 - 0.957344 2024-04-16T15:26:24.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1556 12 1832 1820 0.0730943 0 - 0.957344 2024-04-16T15:26:25.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1557 12 1832 1820 0.0730474 0 - 0.957344 2024-04-16T15:26:26.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1558 12 1832 1820 0.0730005 0 - 0.957344 2024-04-16T15:26:27.697 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1559 12 1832 1820 0.0729537 0 - 0.957344 2024-04-16T15:26:28.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:26:28.700649+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:26:28.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:26:28.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1560 12 1832 1820 0.0729069 0 - 0.957344 2024-04-16T15:26:29.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1561 12 1832 1820 0.0728602 0 - 0.957344 2024-04-16T15:26:30.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1562 12 1832 1820 0.0728136 0 - 0.957344 2024-04-16T15:26:31.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1563 12 1832 1820 0.072767 0 - 0.957344 2024-04-16T15:26:32.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1564 12 1832 1820 0.0727205 0 - 0.957344 2024-04-16T15:26:33.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1565 12 1832 1820 0.072674 0 - 0.957344 2024-04-16T15:26:34.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1566 12 1832 1820 0.0726276 0 - 0.957344 2024-04-16T15:26:35.698 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1567 12 1832 1820 0.0725812 0 - 0.957344 2024-04-16T15:26:36.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1568 12 1832 1820 0.0725349 0 - 0.957344 2024-04-16T15:26:37.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1569 12 1832 1820 0.0724887 0 - 0.957344 2024-04-16T15:26:38.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1570 12 1832 1820 0.0724425 0 - 0.957344 2024-04-16T15:26:39.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1571 12 1832 1820 0.0723964 0 - 0.957344 2024-04-16T15:26:40.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1572 12 1832 1820 0.0723504 0 - 0.957344 2024-04-16T15:26:41.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1573 12 1832 1820 0.0723044 0 - 0.957344 2024-04-16T15:26:42.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1574 12 1832 1820 0.0722584 0 - 0.957344 2024-04-16T15:26:43.699 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1575 12 1832 1820 0.0722126 0 - 0.957344 2024-04-16T15:26:44.700 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1576 12 1832 1820 0.0721668 0 - 0.957344 2024-04-16T15:26:45.700 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1577 12 1832 1820 0.072121 0 - 0.957344 2024-04-16T15:26:46.700 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1578 12 1832 1820 0.0720753 0 - 0.957344 2024-04-16T15:26:47.700 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1579 12 1832 1820 0.0720296 0 - 0.957344 2024-04-16T15:26:48.700 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:26:48.703282+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:26:48.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:26:48.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1580 12 1832 1820 0.071984 0 - 0.957344 2024-04-16T15:26:49.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1581 12 1832 1820 0.0719385 0 - 0.957344 2024-04-16T15:26:50.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1582 12 1832 1820 0.071893 0 - 0.957344 2024-04-16T15:26:51.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1583 12 1832 1820 0.0718476 0 - 0.957344 2024-04-16T15:26:52.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1584 12 1832 1820 0.0718023 0 - 0.957344 2024-04-16T15:26:53.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1585 12 1832 1820 0.071757 0 - 0.957344 2024-04-16T15:26:54.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1586 12 1832 1820 0.0717117 0 - 0.957344 2024-04-16T15:26:55.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1587 12 1832 1820 0.0716665 0 - 0.957344 2024-04-16T15:26:56.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1588 12 1832 1820 0.0716214 0 - 0.957344 2024-04-16T15:26:57.701 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1589 12 1832 1820 0.0715763 0 - 0.957344 2024-04-16T15:26:58.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1590 12 1832 1820 0.0715313 0 - 0.957344 2024-04-16T15:26:59.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1591 12 1832 1820 0.0714864 0 - 0.957344 2024-04-16T15:27:00.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1592 12 1832 1820 0.0714415 0 - 0.957344 2024-04-16T15:27:01.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1593 12 1832 1820 0.0713966 0 - 0.957344 2024-04-16T15:27:02.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1594 12 1832 1820 0.0713518 0 - 0.957344 2024-04-16T15:27:03.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1595 12 1832 1820 0.0713071 0 - 0.957344 2024-04-16T15:27:04.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1596 12 1832 1820 0.0712624 0 - 0.957344 2024-04-16T15:27:05.702 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1597 12 1832 1820 0.0712178 0 - 0.957344 2024-04-16T15:27:06.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1598 12 1832 1820 0.0711732 0 - 0.957344 2024-04-16T15:27:07.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1599 12 1832 1820 0.0711287 0 - 0.957344 2024-04-16T15:27:08.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:27:08.705984+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:27:08.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:27:08.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1600 12 1832 1820 0.0710842 0 - 0.957344 2024-04-16T15:27:09.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1601 12 1832 1820 0.0710398 0 - 0.957344 2024-04-16T15:27:10.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1602 12 1832 1820 0.0709955 0 - 0.957344 2024-04-16T15:27:11.703 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1603 12 1832 1820 0.0709512 0 - 0.957344 2024-04-16T15:27:12.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1604 12 1832 1820 0.070907 0 - 0.957344 2024-04-16T15:27:13.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1605 12 1832 1820 0.0708628 0 - 0.957344 2024-04-16T15:27:14.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1606 12 1832 1820 0.0708187 0 - 0.957344 2024-04-16T15:27:15.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1607 12 1832 1820 0.0707746 0 - 0.957344 2024-04-16T15:27:16.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1608 12 1832 1820 0.0707306 0 - 0.957344 2024-04-16T15:27:17.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1609 12 1832 1820 0.0706866 0 - 0.957344 2024-04-16T15:27:18.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1610 12 1832 1820 0.0706427 0 - 0.957344 2024-04-16T15:27:19.704 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1611 12 1832 1820 0.0705989 0 - 0.957344 2024-04-16T15:27:20.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1612 12 1832 1820 0.0705551 0 - 0.957344 2024-04-16T15:27:21.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1613 12 1832 1820 0.0705113 0 - 0.957344 2024-04-16T15:27:22.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1614 12 1832 1820 0.0704677 0 - 0.957344 2024-04-16T15:27:23.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1615 12 1832 1820 0.070424 0 - 0.957344 2024-04-16T15:27:24.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1616 12 1832 1820 0.0703804 0 - 0.957344 2024-04-16T15:27:25.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1617 12 1832 1820 0.0703369 0 - 0.957344 2024-04-16T15:27:26.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1618 12 1832 1820 0.0702934 0 - 0.957344 2024-04-16T15:27:27.705 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1619 12 1832 1820 0.07025 0 - 0.957344 2024-04-16T15:27:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:27:28.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:27:28.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:27:28.708631+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:27:28.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:27:28.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1620 12 1832 1820 0.0702067 0 - 0.957344 2024-04-16T15:27:29.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1621 12 1832 1820 0.0701634 0 - 0.957344 2024-04-16T15:27:30.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:27:30.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:27:30.428+0000 7f9b04c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:27:30.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1622 12 1832 1820 0.0701201 0 - 0.957344 2024-04-16T15:27:31.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1623 12 1832 1820 0.0700769 0 - 0.957344 2024-04-16T15:27:32.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1624 12 1832 1820 0.0700337 0 - 0.957344 2024-04-16T15:27:33.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1625 12 1832 1820 0.0699906 0 - 0.957344 2024-04-16T15:27:34.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1626 12 1832 1820 0.0699476 0 - 0.957344 2024-04-16T15:27:35.706 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1627 12 1832 1820 0.0699046 0 - 0.957344 2024-04-16T15:27:36.707 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1628 12 1832 1820 0.0698617 0 - 0.957344 2024-04-16T15:27:37.707 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1629 12 1832 1820 0.0698188 0 - 0.957344 2024-04-16T15:27:38.707 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1630 12 1832 1820 0.0697759 0 - 0.957344 2024-04-16T15:27:39.707 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1631 12 1832 1820 0.0697332 0 - 0.957344 2024-04-16T15:27:40.707 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1632 12 1832 1820 0.0696904 0 - 0.957344 2024-04-16T15:27:41.707 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1633 12 1832 1820 0.0696478 0 - 0.957344 2024-04-16T15:27:42.708 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1634 12 1832 1820 0.0696051 0 - 0.957344 2024-04-16T15:27:43.708 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1635 12 1832 1820 0.0695626 0 - 0.957344 2024-04-16T15:27:44.708 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1636 12 1832 1820 0.06952 0 - 0.957344 2024-04-16T15:27:45.708 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1637 12 1832 1820 0.0694776 0 - 0.957344 2024-04-16T15:27:46.708 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1638 12 1832 1820 0.0694352 0 - 0.957344 2024-04-16T15:27:47.708 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1639 12 1832 1820 0.0693928 0 - 0.957344 2024-04-16T15:27:48.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:27:48.711439+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:27:48.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:27:48.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1640 12 1832 1820 0.0693505 0 - 0.957344 2024-04-16T15:27:49.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1641 12 1832 1820 0.0693082 0 - 0.957344 2024-04-16T15:27:50.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1642 12 1832 1820 0.069266 0 - 0.957344 2024-04-16T15:27:51.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1643 12 1832 1820 0.0692238 0 - 0.957344 2024-04-16T15:27:52.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1644 12 1832 1820 0.0691817 0 - 0.957344 2024-04-16T15:27:53.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1645 12 1832 1820 0.0691397 0 - 0.957344 2024-04-16T15:27:54.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1646 12 1832 1820 0.0690977 0 - 0.957344 2024-04-16T15:27:55.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1647 12 1832 1820 0.0690557 0 - 0.957344 2024-04-16T15:27:56.709 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1648 12 1832 1820 0.0690138 0 - 0.957344 2024-04-16T15:27:57.710 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1649 12 1832 1820 0.068972 0 - 0.957344 2024-04-16T15:27:58.710 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1650 12 1832 1820 0.0689302 0 - 0.957344 2024-04-16T15:27:59.710 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1651 12 1832 1820 0.0688884 0 - 0.957344 2024-04-16T15:28:00.710 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1652 12 1832 1820 0.0688467 0 - 0.957344 2024-04-16T15:28:01.710 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1653 12 1832 1820 0.0688051 0 - 0.957344 2024-04-16T15:28:02.710 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1654 12 1832 1820 0.0687635 0 - 0.957344 2024-04-16T15:28:03.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1655 12 1832 1820 0.0687219 0 - 0.957344 2024-04-16T15:28:04.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1656 12 1832 1820 0.0686804 0 - 0.957344 2024-04-16T15:28:05.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1657 12 1832 1820 0.068639 0 - 0.957344 2024-04-16T15:28:06.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1658 12 1832 1820 0.0685976 0 - 0.957344 2024-04-16T15:28:07.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1659 12 1832 1820 0.0685562 0 - 0.957344 2024-04-16T15:28:08.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:28:08.714190+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:28:08.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:28:08.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1660 12 1832 1820 0.0685149 0 - 0.957344 2024-04-16T15:28:09.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1661 12 1832 1820 0.0684737 0 - 0.957344 2024-04-16T15:28:10.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1662 12 1832 1820 0.0684325 0 - 0.957344 2024-04-16T15:28:11.711 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1663 12 1832 1820 0.0683913 0 - 0.957344 2024-04-16T15:28:12.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1664 12 1832 1820 0.0683502 0 - 0.957344 2024-04-16T15:28:13.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1665 12 1832 1820 0.0683092 0 - 0.957344 2024-04-16T15:28:14.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1666 12 1832 1820 0.0682682 0 - 0.957344 2024-04-16T15:28:15.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1667 12 1832 1820 0.0682272 0 - 0.957344 2024-04-16T15:28:16.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1668 12 1832 1820 0.0681863 0 - 0.957344 2024-04-16T15:28:17.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1669 12 1832 1820 0.0681455 0 - 0.957344 2024-04-16T15:28:18.712 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1670 12 1832 1820 0.0681047 0 - 0.957344 2024-04-16T15:28:19.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1671 12 1832 1820 0.0680639 0 - 0.957344 2024-04-16T15:28:20.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1672 12 1832 1820 0.0680232 0 - 0.957344 2024-04-16T15:28:21.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1673 12 1832 1820 0.0679825 0 - 0.957344 2024-04-16T15:28:22.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1674 12 1832 1820 0.0679419 0 - 0.957344 2024-04-16T15:28:23.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1675 12 1832 1820 0.0679014 0 - 0.957344 2024-04-16T15:28:24.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1676 12 1832 1820 0.0678608 0 - 0.957344 2024-04-16T15:28:25.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1677 12 1832 1820 0.0678204 0 - 0.957344 2024-04-16T15:28:26.713 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1678 12 1832 1820 0.06778 0 - 0.957344 2024-04-16T15:28:27.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1679 12 1832 1820 0.0677396 0 - 0.957344 2024-04-16T15:28:28.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:28:28.716820+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:28:28.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:28:28.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1680 12 1832 1820 0.0676993 0 - 0.957344 2024-04-16T15:28:29.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1681 12 1832 1820 0.067659 0 - 0.957344 2024-04-16T15:28:30.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1682 12 1832 1820 0.0676188 0 - 0.957344 2024-04-16T15:28:31.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1683 12 1832 1820 0.0675786 0 - 0.957344 2024-04-16T15:28:32.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1684 12 1832 1820 0.0675385 0 - 0.957344 2024-04-16T15:28:33.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1685 12 1832 1820 0.0674984 0 - 0.957344 2024-04-16T15:28:34.714 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1686 12 1832 1820 0.0674584 0 - 0.957344 2024-04-16T15:28:35.715 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1687 12 1832 1820 0.0674184 0 - 0.957344 2024-04-16T15:28:36.715 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1688 12 1832 1820 0.0673784 0 - 0.957344 2024-04-16T15:28:37.715 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1689 12 1832 1820 0.0673385 0 - 0.957344 2024-04-16T15:28:38.715 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1690 12 1832 1820 0.0672987 0 - 0.957344 2024-04-16T15:28:39.715 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1691 12 1832 1820 0.0672589 0 - 0.957344 2024-04-16T15:28:40.715 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1692 12 1832 1820 0.0672191 0 - 0.957344 2024-04-16T15:28:41.716 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1693 12 1832 1820 0.0671794 0 - 0.957344 2024-04-16T15:28:42.716 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1694 12 1832 1820 0.0671398 0 - 0.957344 2024-04-16T15:28:43.716 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1695 12 1832 1820 0.0671002 0 - 0.957344 2024-04-16T15:28:44.716 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1696 12 1832 1820 0.0670606 0 - 0.957344 2024-04-16T15:28:45.716 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1697 12 1832 1820 0.0670211 0 - 0.957344 2024-04-16T15:28:46.716 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1698 12 1832 1820 0.0669816 0 - 0.957344 2024-04-16T15:28:47.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1699 12 1832 1820 0.0669422 0 - 0.957344 2024-04-16T15:28:48.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:28:48.719608+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:28:48.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:28:48.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1700 12 1832 1820 0.0669028 0 - 0.957344 2024-04-16T15:28:49.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1701 12 1832 1820 0.0668635 0 - 0.957344 2024-04-16T15:28:50.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1702 12 1832 1820 0.0668242 0 - 0.957344 2024-04-16T15:28:51.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1703 12 1832 1820 0.066785 0 - 0.957344 2024-04-16T15:28:52.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1704 12 1832 1820 0.0667458 0 - 0.957344 2024-04-16T15:28:53.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1705 12 1832 1820 0.0667066 0 - 0.957344 2024-04-16T15:28:54.717 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1706 12 1832 1820 0.0666675 0 - 0.957344 2024-04-16T15:28:55.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1707 12 1832 1820 0.0666285 0 - 0.957344 2024-04-16T15:28:56.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1708 12 1832 1820 0.0665894 0 - 0.957344 2024-04-16T15:28:57.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1709 12 1832 1820 0.0665505 0 - 0.957344 2024-04-16T15:28:58.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1710 12 1832 1820 0.0665116 0 - 0.957344 2024-04-16T15:28:59.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1711 12 1832 1820 0.0664727 0 - 0.957344 2024-04-16T15:29:00.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1712 12 1832 1820 0.0664339 0 - 0.957344 2024-04-16T15:29:01.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1713 12 1832 1820 0.0663951 0 - 0.957344 2024-04-16T15:29:02.718 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1714 12 1832 1820 0.0663563 0 - 0.957344 2024-04-16T15:29:03.719 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1715 12 1832 1820 0.0663177 0 - 0.957344 2024-04-16T15:29:04.719 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1716 12 1832 1820 0.066279 0 - 0.957344 2024-04-16T15:29:05.719 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1717 12 1832 1820 0.0662404 0 - 0.957344 2024-04-16T15:29:06.719 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1718 12 1832 1820 0.0662018 0 - 0.957344 2024-04-16T15:29:07.719 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1719 12 1832 1820 0.0661633 0 - 0.957344 2024-04-16T15:29:08.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:29:08.722426+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:29:08.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:29:08.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1720 12 1832 1820 0.0661249 0 - 0.957344 2024-04-16T15:29:09.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1721 12 1832 1820 0.0660864 0 - 0.957344 2024-04-16T15:29:10.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1722 12 1832 1820 0.0660481 0 - 0.957344 2024-04-16T15:29:11.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1723 12 1832 1820 0.0660097 0 - 0.957344 2024-04-16T15:29:12.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1724 12 1832 1820 0.0659714 0 - 0.957344 2024-04-16T15:29:13.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1725 12 1832 1820 0.0659332 0 - 0.957344 2024-04-16T15:29:14.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1726 12 1832 1820 0.065895 0 - 0.957344 2024-04-16T15:29:15.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1727 12 1832 1820 0.0658568 0 - 0.957344 2024-04-16T15:29:16.720 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1728 12 1832 1820 0.0658187 0 - 0.957344 2024-04-16T15:29:17.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1729 12 1832 1820 0.0657807 0 - 0.957344 2024-04-16T15:29:18.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1730 12 1832 1820 0.0657426 0 - 0.957344 2024-04-16T15:29:19.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1731 12 1832 1820 0.0657047 0 - 0.957344 2024-04-16T15:29:20.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1732 12 1832 1820 0.0656667 0 - 0.957344 2024-04-16T15:29:21.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1733 12 1832 1820 0.0656288 0 - 0.957344 2024-04-16T15:29:22.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1734 12 1832 1820 0.065591 0 - 0.957344 2024-04-16T15:29:23.721 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1735 12 1832 1820 0.0655532 0 - 0.957344 2024-04-16T15:29:24.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1736 12 1832 1820 0.0655154 0 - 0.957344 2024-04-16T15:29:25.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1737 12 1832 1820 0.0654777 0 - 0.957344 2024-04-16T15:29:26.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1738 12 1832 1820 0.06544 0 - 0.957344 2024-04-16T15:29:27.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1739 12 1832 1820 0.0654024 0 - 0.957344 2024-04-16T15:29:28.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:29:28.725079+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:29:28.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:29:28.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1740 12 1832 1820 0.0653648 0 - 0.957344 2024-04-16T15:29:29.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1741 12 1832 1820 0.0653273 0 - 0.957344 2024-04-16T15:29:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:29:30.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:29:30.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1742 12 1832 1820 0.0652898 0 - 0.957344 2024-04-16T15:29:31.722 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1743 12 1832 1820 0.0652523 0 - 0.957344 2024-04-16T15:29:32.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:29:32.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:29:32.469+0000 7f89f38ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:29:32.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1744 12 1832 1820 0.0652149 0 - 0.957344 2024-04-16T15:29:33.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1745 12 1832 1820 0.0651775 0 - 0.957344 2024-04-16T15:29:34.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1746 12 1832 1820 0.0651402 0 - 0.957344 2024-04-16T15:29:35.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1747 12 1832 1820 0.0651029 0 - 0.957344 2024-04-16T15:29:36.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1748 12 1832 1820 0.0650657 0 - 0.957344 2024-04-16T15:29:37.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1749 12 1832 1820 0.0650285 0 - 0.957344 2024-04-16T15:29:38.723 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1750 12 1832 1820 0.0649913 0 - 0.957344 2024-04-16T15:29:39.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1751 12 1832 1820 0.0649542 0 - 0.957344 2024-04-16T15:29:40.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1752 12 1832 1820 0.0649171 0 - 0.957344 2024-04-16T15:29:41.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1753 12 1832 1820 0.0648801 0 - 0.957344 2024-04-16T15:29:42.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1754 12 1832 1820 0.0648431 0 - 0.957344 2024-04-16T15:29:43.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1755 12 1832 1820 0.0648061 0 - 0.957344 2024-04-16T15:29:44.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1756 12 1832 1820 0.0647692 0 - 0.957344 2024-04-16T15:29:45.724 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1757 12 1832 1820 0.0647324 0 - 0.957344 2024-04-16T15:29:46.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1758 12 1832 1820 0.0646955 0 - 0.957344 2024-04-16T15:29:47.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1759 12 1832 1820 0.0646588 0 - 0.957344 2024-04-16T15:29:48.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:29:48.727813+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:29:48.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:29:48.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1760 12 1832 1820 0.064622 0 - 0.957344 2024-04-16T15:29:49.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1761 12 1832 1820 0.0645853 0 - 0.957344 2024-04-16T15:29:50.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1762 12 1832 1820 0.0645487 0 - 0.957344 2024-04-16T15:29:51.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1763 12 1832 1820 0.0645121 0 - 0.957344 2024-04-16T15:29:52.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1764 12 1832 1820 0.0644755 0 - 0.957344 2024-04-16T15:29:53.725 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1765 12 1832 1820 0.064439 0 - 0.957344 2024-04-16T15:29:54.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1766 12 1832 1820 0.0644025 0 - 0.957344 2024-04-16T15:29:55.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1767 12 1832 1820 0.064366 0 - 0.957344 2024-04-16T15:29:56.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1768 12 1832 1820 0.0643296 0 - 0.957344 2024-04-16T15:29:57.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1769 12 1832 1820 0.0642933 0 - 0.957344 2024-04-16T15:29:58.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1770 12 1832 1820 0.0642569 0 - 0.957344 2024-04-16T15:29:59.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1771 12 1832 1820 0.0642206 0 - 0.957344 2024-04-16T15:30:00.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1772 12 1832 1820 0.0641844 0 - 0.957344 2024-04-16T15:30:01.726 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1773 12 1832 1820 0.0641482 0 - 0.957344 2024-04-16T15:30:02.727 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1774 12 1832 1820 0.064112 0 - 0.957344 2024-04-16T15:30:03.727 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1775 12 1832 1820 0.0640759 0 - 0.957344 2024-04-16T15:30:04.727 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1776 12 1832 1820 0.0640398 0 - 0.957344 2024-04-16T15:30:05.727 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1777 12 1832 1820 0.0640038 0 - 0.957344 2024-04-16T15:30:06.727 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1778 12 1832 1820 0.0639678 0 - 0.957344 2024-04-16T15:30:07.727 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1779 12 1832 1820 0.0639319 0 - 0.957344 2024-04-16T15:30:08.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:30:08.730448+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:30:08.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:30:08.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1780 12 1832 1820 0.0638959 0 - 0.957344 2024-04-16T15:30:09.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1781 12 1832 1820 0.0638601 0 - 0.957344 2024-04-16T15:30:10.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1782 12 1832 1820 0.0638242 0 - 0.957344 2024-04-16T15:30:11.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1783 12 1832 1820 0.0637884 0 - 0.957344 2024-04-16T15:30:12.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1784 12 1832 1820 0.0637527 0 - 0.957344 2024-04-16T15:30:13.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1785 12 1832 1820 0.063717 0 - 0.957344 2024-04-16T15:30:14.728 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1786 12 1832 1820 0.0636813 0 - 0.957344 2024-04-16T15:30:15.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1787 12 1832 1820 0.0636456 0 - 0.957344 2024-04-16T15:30:16.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1788 12 1832 1820 0.06361 0 - 0.957344 2024-04-16T15:30:17.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1789 12 1832 1820 0.0635745 0 - 0.957344 2024-04-16T15:30:18.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1790 12 1832 1820 0.063539 0 - 0.957344 2024-04-16T15:30:19.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1791 12 1832 1820 0.0635035 0 - 0.957344 2024-04-16T15:30:20.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1792 12 1832 1820 0.0634681 0 - 0.957344 2024-04-16T15:30:21.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1793 12 1832 1820 0.0634327 0 - 0.957344 2024-04-16T15:30:22.729 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1794 12 1832 1820 0.0633973 0 - 0.957344 2024-04-16T15:30:23.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1795 12 1832 1820 0.063362 0 - 0.957344 2024-04-16T15:30:24.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1796 12 1832 1820 0.0633267 0 - 0.957344 2024-04-16T15:30:25.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1797 12 1832 1820 0.0632915 0 - 0.957344 2024-04-16T15:30:26.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1798 12 1832 1820 0.0632563 0 - 0.957344 2024-04-16T15:30:27.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1799 12 1832 1820 0.0632211 0 - 0.957344 2024-04-16T15:30:28.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:30:28.733327+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:30:28.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:30:28.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1800 12 1832 1820 0.063186 0 - 0.957344 2024-04-16T15:30:29.730 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1801 12 1832 1820 0.0631509 0 - 0.957344 2024-04-16T15:30:30.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1802 12 1832 1820 0.0631158 0 - 0.957344 2024-04-16T15:30:31.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1803 12 1832 1820 0.0630808 0 - 0.957344 2024-04-16T15:30:32.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1804 12 1832 1820 0.0630459 0 - 0.957344 2024-04-16T15:30:33.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1805 12 1832 1820 0.0630109 0 - 0.957344 2024-04-16T15:30:34.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1806 12 1832 1820 0.0629761 0 - 0.957344 2024-04-16T15:30:35.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1807 12 1832 1820 0.0629412 0 - 0.957344 2024-04-16T15:30:36.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1808 12 1832 1820 0.0629064 0 - 0.957344 2024-04-16T15:30:37.731 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1809 12 1832 1820 0.0628716 0 - 0.957344 2024-04-16T15:30:38.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1810 12 1832 1820 0.0628369 0 - 0.957344 2024-04-16T15:30:39.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1811 12 1832 1820 0.0628022 0 - 0.957344 2024-04-16T15:30:40.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1812 12 1832 1820 0.0627675 0 - 0.957344 2024-04-16T15:30:41.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1813 12 1832 1820 0.0627329 0 - 0.957344 2024-04-16T15:30:42.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1814 12 1832 1820 0.0626983 0 - 0.957344 2024-04-16T15:30:43.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1815 12 1832 1820 0.0626638 0 - 0.957344 2024-04-16T15:30:44.732 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1816 12 1832 1820 0.0626293 0 - 0.957344 2024-04-16T15:30:45.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1817 12 1832 1820 0.0625948 0 - 0.957344 2024-04-16T15:30:46.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1818 12 1832 1820 0.0625604 0 - 0.957344 2024-04-16T15:30:47.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1819 12 1832 1820 0.062526 0 - 0.957344 2024-04-16T15:30:48.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:30:48.735985+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:30:48.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:30:48.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1820 12 1832 1820 0.0624916 0 - 0.957344 2024-04-16T15:30:49.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1821 12 1832 1820 0.0624573 0 - 0.957344 2024-04-16T15:30:50.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1822 12 1832 1820 0.062423 0 - 0.957344 2024-04-16T15:30:51.733 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1823 12 1832 1820 0.0623888 0 - 0.957344 2024-04-16T15:30:52.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1824 12 1832 1820 0.0623546 0 - 0.957344 2024-04-16T15:30:53.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1825 12 1832 1820 0.0623204 0 - 0.957344 2024-04-16T15:30:54.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1826 12 1832 1820 0.0622863 0 - 0.957344 2024-04-16T15:30:55.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1827 12 1832 1820 0.0622522 0 - 0.957344 2024-04-16T15:30:56.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1828 12 1832 1820 0.0622181 0 - 0.957344 2024-04-16T15:30:57.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1829 12 1832 1820 0.0621841 0 - 0.957344 2024-04-16T15:30:58.734 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1830 12 1832 1820 0.0621501 0 - 0.957344 2024-04-16T15:30:59.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1831 12 1832 1820 0.0621162 0 - 0.957344 2024-04-16T15:31:00.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1832 12 1832 1820 0.0620823 0 - 0.957344 2024-04-16T15:31:01.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1833 12 1832 1820 0.0620484 0 - 0.957344 2024-04-16T15:31:02.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1834 12 1832 1820 0.0620146 0 - 0.957344 2024-04-16T15:31:03.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1835 12 1832 1820 0.0619808 0 - 0.957344 2024-04-16T15:31:04.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1836 12 1832 1820 0.061947 0 - 0.957344 2024-04-16T15:31:05.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1837 12 1832 1820 0.0619133 0 - 0.957344 2024-04-16T15:31:06.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1838 12 1832 1820 0.0618796 0 - 0.957344 2024-04-16T15:31:07.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1839 12 1832 1820 0.061846 0 - 0.957344 2024-04-16T15:31:08.735 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:31:08.738412+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:31:08.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:31:08.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1840 12 1832 1820 0.0618124 0 - 0.957344 2024-04-16T15:31:09.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1841 12 1832 1820 0.0617788 0 - 0.957344 2024-04-16T15:31:10.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1842 12 1832 1820 0.0617453 0 - 0.957344 2024-04-16T15:31:11.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1843 12 1832 1820 0.0617118 0 - 0.957344 2024-04-16T15:31:12.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1844 12 1832 1820 0.0616783 0 - 0.957344 2024-04-16T15:31:13.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1845 12 1832 1820 0.0616449 0 - 0.957344 2024-04-16T15:31:14.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1846 12 1832 1820 0.0616115 0 - 0.957344 2024-04-16T15:31:15.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1847 12 1832 1820 0.0615781 0 - 0.957344 2024-04-16T15:31:16.736 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1848 12 1832 1820 0.0615448 0 - 0.957344 2024-04-16T15:31:17.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1849 12 1832 1820 0.0615115 0 - 0.957344 2024-04-16T15:31:18.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1850 12 1832 1820 0.0614783 0 - 0.957344 2024-04-16T15:31:19.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1851 12 1832 1820 0.061445 0 - 0.957344 2024-04-16T15:31:20.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1852 12 1832 1820 0.0614119 0 - 0.957344 2024-04-16T15:31:21.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1853 12 1832 1820 0.0613787 0 - 0.957344 2024-04-16T15:31:22.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1854 12 1832 1820 0.0613456 0 - 0.957344 2024-04-16T15:31:23.737 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1855 12 1832 1820 0.0613125 0 - 0.957344 2024-04-16T15:31:24.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1856 12 1832 1820 0.0612795 0 - 0.957344 2024-04-16T15:31:25.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1857 12 1832 1820 0.0612465 0 - 0.957344 2024-04-16T15:31:26.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1858 12 1832 1820 0.0612135 0 - 0.957344 2024-04-16T15:31:27.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1859 12 1832 1820 0.0611806 0 - 0.957344 2024-04-16T15:31:28.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:31:28.741210+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:31:28.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:31:28.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1860 12 1832 1820 0.0611477 0 - 0.957344 2024-04-16T15:31:29.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1861 12 1832 1820 0.0611149 0 - 0.957344 2024-04-16T15:31:30.738 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1862 12 1832 1820 0.061082 0 - 0.957344 2024-04-16T15:31:31.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1863 12 1832 1820 0.0610493 0 - 0.957344 2024-04-16T15:31:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:31:32.387 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:31:32.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1864 12 1832 1820 0.0610165 0 - 0.957344 2024-04-16T15:31:33.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1865 12 1832 1820 0.0609838 0 - 0.957344 2024-04-16T15:31:34.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:31:34.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:31:34.511+0000 7fb445f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:31:34.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1866 12 1832 1820 0.0609511 0 - 0.957344 2024-04-16T15:31:35.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1867 12 1832 1820 0.0609185 0 - 0.957344 2024-04-16T15:31:36.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1868 12 1832 1820 0.0608858 0 - 0.957344 2024-04-16T15:31:37.739 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1869 12 1832 1820 0.0608533 0 - 0.957344 2024-04-16T15:31:38.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1870 12 1832 1820 0.0608207 0 - 0.957344 2024-04-16T15:31:39.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1871 12 1832 1820 0.0607882 0 - 0.957344 2024-04-16T15:31:40.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1872 12 1832 1820 0.0607557 0 - 0.957344 2024-04-16T15:31:41.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1873 12 1832 1820 0.0607233 0 - 0.957344 2024-04-16T15:31:42.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1874 12 1832 1820 0.0606909 0 - 0.957344 2024-04-16T15:31:43.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1875 12 1832 1820 0.0606585 0 - 0.957344 2024-04-16T15:31:44.740 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1876 12 1832 1820 0.0606262 0 - 0.957344 2024-04-16T15:31:45.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1877 12 1832 1820 0.0605939 0 - 0.957344 2024-04-16T15:31:46.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1878 12 1832 1820 0.0605616 0 - 0.957344 2024-04-16T15:31:47.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1879 12 1832 1820 0.0605294 0 - 0.957344 2024-04-16T15:31:48.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:31:48.743893+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:31:48.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:31:48.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1880 12 1832 1820 0.0604972 0 - 0.957344 2024-04-16T15:31:49.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1881 12 1832 1820 0.0604651 0 - 0.957344 2024-04-16T15:31:50.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1882 12 1832 1820 0.0604329 0 - 0.957344 2024-04-16T15:31:51.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1883 12 1832 1820 0.0604008 0 - 0.957344 2024-04-16T15:31:52.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1884 12 1832 1820 0.0603688 0 - 0.957344 2024-04-16T15:31:53.741 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1885 12 1832 1820 0.0603367 0 - 0.957344 2024-04-16T15:31:54.742 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1886 12 1832 1820 0.0603048 0 - 0.957344 2024-04-16T15:31:55.742 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1887 12 1832 1820 0.0602728 0 - 0.957344 2024-04-16T15:31:56.742 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1888 12 1832 1820 0.0602409 0 - 0.957344 2024-04-16T15:31:57.742 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1889 12 1832 1820 0.060209 0 - 0.957344 2024-04-16T15:31:58.742 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1890 12 1832 1820 0.0601771 0 - 0.957344 2024-04-16T15:31:59.742 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1891 12 1832 1820 0.0601453 0 - 0.957344 2024-04-16T15:32:00.743 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1892 12 1832 1820 0.0601135 0 - 0.957344 2024-04-16T15:32:01.743 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1893 12 1832 1820 0.0600818 0 - 0.957344 2024-04-16T15:32:02.743 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1894 12 1832 1820 0.06005 0 - 0.957344 2024-04-16T15:32:03.743 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1895 12 1832 1820 0.0600183 0 - 0.957344 2024-04-16T15:32:04.743 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1896 12 1832 1820 0.0599867 0 - 0.957344 2024-04-16T15:32:05.743 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1897 12 1832 1820 0.0599551 0 - 0.957344 2024-04-16T15:32:06.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1898 12 1832 1820 0.0599235 0 - 0.957344 2024-04-16T15:32:07.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1899 12 1832 1820 0.0598919 0 - 0.957344 2024-04-16T15:32:08.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:32:08.746742+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:32:08.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:32:08.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1900 12 1832 1820 0.0598604 0 - 0.957344 2024-04-16T15:32:09.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1901 12 1832 1820 0.0598289 0 - 0.957344 2024-04-16T15:32:10.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1902 12 1832 1820 0.0597975 0 - 0.957344 2024-04-16T15:32:11.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1903 12 1832 1820 0.059766 0 - 0.957344 2024-04-16T15:32:12.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1904 12 1832 1820 0.0597346 0 - 0.957344 2024-04-16T15:32:13.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1905 12 1832 1820 0.0597033 0 - 0.957344 2024-04-16T15:32:14.744 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1906 12 1832 1820 0.059672 0 - 0.957344 2024-04-16T15:32:15.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1907 12 1832 1820 0.0596407 0 - 0.957344 2024-04-16T15:32:16.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1908 12 1832 1820 0.0596094 0 - 0.957344 2024-04-16T15:32:17.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1909 12 1832 1820 0.0595782 0 - 0.957344 2024-04-16T15:32:18.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1910 12 1832 1820 0.059547 0 - 0.957344 2024-04-16T15:32:19.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1911 12 1832 1820 0.0595158 0 - 0.957344 2024-04-16T15:32:20.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1912 12 1832 1820 0.0594847 0 - 0.957344 2024-04-16T15:32:21.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1913 12 1832 1820 0.0594536 0 - 0.957344 2024-04-16T15:32:22.745 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1914 12 1832 1820 0.0594226 0 - 0.957344 2024-04-16T15:32:23.746 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1915 12 1832 1820 0.0593915 0 - 0.957344 2024-04-16T15:32:24.746 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1916 12 1832 1820 0.0593605 0 - 0.957344 2024-04-16T15:32:25.746 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1917 12 1832 1820 0.0593296 0 - 0.957344 2024-04-16T15:32:26.746 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1918 12 1832 1820 0.0592986 0 - 0.957344 2024-04-16T15:32:27.746 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1919 12 1832 1820 0.0592677 0 - 0.957344 2024-04-16T15:32:28.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:32:28.749323+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:32:28.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:32:28.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1920 12 1832 1820 0.0592369 0 - 0.957344 2024-04-16T15:32:29.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1921 12 1832 1820 0.059206 0 - 0.957344 2024-04-16T15:32:30.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1922 12 1832 1820 0.0591752 0 - 0.957344 2024-04-16T15:32:31.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1923 12 1832 1820 0.0591444 0 - 0.957344 2024-04-16T15:32:32.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1924 12 1832 1820 0.0591137 0 - 0.957344 2024-04-16T15:32:33.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1925 12 1832 1820 0.059083 0 - 0.957344 2024-04-16T15:32:34.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1926 12 1832 1820 0.0590523 0 - 0.957344 2024-04-16T15:32:35.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1927 12 1832 1820 0.0590217 0 - 0.957344 2024-04-16T15:32:36.747 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1928 12 1832 1820 0.0589911 0 - 0.957344 2024-04-16T15:32:37.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1929 12 1832 1820 0.0589605 0 - 0.957344 2024-04-16T15:32:38.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1930 12 1832 1820 0.0589299 0 - 0.957344 2024-04-16T15:32:39.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1931 12 1832 1820 0.0588994 0 - 0.957344 2024-04-16T15:32:40.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1932 12 1832 1820 0.0588689 0 - 0.957344 2024-04-16T15:32:41.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1933 12 1832 1820 0.0588385 0 - 0.957344 2024-04-16T15:32:42.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1934 12 1832 1820 0.058808 0 - 0.957344 2024-04-16T15:32:43.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1935 12 1832 1820 0.0587777 0 - 0.957344 2024-04-16T15:32:44.748 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1936 12 1832 1820 0.0587473 0 - 0.957344 2024-04-16T15:32:45.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1937 12 1832 1820 0.058717 0 - 0.957344 2024-04-16T15:32:46.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1938 12 1832 1820 0.0586867 0 - 0.957344 2024-04-16T15:32:47.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1939 12 1832 1820 0.0586564 0 - 0.957344 2024-04-16T15:32:48.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:32:48.751998+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:32:48.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:32:48.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1940 12 1832 1820 0.0586262 0 - 0.957344 2024-04-16T15:32:49.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1941 12 1832 1820 0.058596 0 - 0.957344 2024-04-16T15:32:50.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1942 12 1832 1820 0.0585658 0 - 0.957344 2024-04-16T15:32:51.749 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1943 12 1832 1820 0.0585356 0 - 0.957344 2024-04-16T15:32:52.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1944 12 1832 1820 0.0585055 0 - 0.957344 2024-04-16T15:32:53.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1945 12 1832 1820 0.0584755 0 - 0.957344 2024-04-16T15:32:54.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1946 12 1832 1820 0.0584454 0 - 0.957344 2024-04-16T15:32:55.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1947 12 1832 1820 0.0584154 0 - 0.957344 2024-04-16T15:32:56.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1948 12 1832 1820 0.0583854 0 - 0.957344 2024-04-16T15:32:57.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1949 12 1832 1820 0.0583554 0 - 0.957344 2024-04-16T15:32:58.750 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1950 12 1832 1820 0.0583255 0 - 0.957344 2024-04-16T15:32:59.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1951 12 1832 1820 0.0582956 0 - 0.957344 2024-04-16T15:33:00.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1952 12 1832 1820 0.0582658 0 - 0.957344 2024-04-16T15:33:01.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1953 12 1832 1820 0.0582359 0 - 0.957344 2024-04-16T15:33:02.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1954 12 1832 1820 0.0582061 0 - 0.957344 2024-04-16T15:33:03.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1955 12 1832 1820 0.0581763 0 - 0.957344 2024-04-16T15:33:04.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1956 12 1832 1820 0.0581466 0 - 0.957344 2024-04-16T15:33:05.751 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1957 12 1832 1820 0.0581169 0 - 0.957344 2024-04-16T15:33:06.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1958 12 1832 1820 0.0580872 0 - 0.957344 2024-04-16T15:33:07.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1959 12 1832 1820 0.0580576 0 - 0.957344 2024-04-16T15:33:08.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:33:08.754690+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:33:08.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:33:08.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1960 12 1832 1820 0.0580279 0 - 0.957344 2024-04-16T15:33:09.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1961 12 1832 1820 0.0579983 0 - 0.957344 2024-04-16T15:33:10.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1962 12 1832 1820 0.0579688 0 - 0.957344 2024-04-16T15:33:11.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1963 12 1832 1820 0.0579393 0 - 0.957344 2024-04-16T15:33:12.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1964 12 1832 1820 0.0579098 0 - 0.957344 2024-04-16T15:33:13.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1965 12 1832 1820 0.0578803 0 - 0.957344 2024-04-16T15:33:14.752 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1966 12 1832 1820 0.0578508 0 - 0.957344 2024-04-16T15:33:15.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1967 12 1832 1820 0.0578214 0 - 0.957344 2024-04-16T15:33:16.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1968 12 1832 1820 0.0577921 0 - 0.957344 2024-04-16T15:33:17.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1969 12 1832 1820 0.0577627 0 - 0.957344 2024-04-16T15:33:18.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1970 12 1832 1820 0.0577334 0 - 0.957344 2024-04-16T15:33:19.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1971 12 1832 1820 0.0577041 0 - 0.957344 2024-04-16T15:33:20.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1972 12 1832 1820 0.0576748 0 - 0.957344 2024-04-16T15:33:21.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1973 12 1832 1820 0.0576456 0 - 0.957344 2024-04-16T15:33:22.753 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1974 12 1832 1820 0.0576164 0 - 0.957344 2024-04-16T15:33:23.754 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1975 12 1832 1820 0.0575872 0 - 0.957344 2024-04-16T15:33:24.754 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1976 12 1832 1820 0.0575581 0 - 0.957344 2024-04-16T15:33:25.754 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1977 12 1832 1820 0.057529 0 - 0.957344 2024-04-16T15:33:26.754 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1978 12 1832 1820 0.0574999 0 - 0.957344 2024-04-16T15:33:27.754 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1979 12 1832 1820 0.0574708 0 - 0.957344 2024-04-16T15:33:28.754 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:33:28.757306+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:33:28.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:33:28.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1980 12 1832 1820 0.0574418 0 - 0.957344 2024-04-16T15:33:29.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1981 12 1832 1820 0.0574128 0 - 0.957344 2024-04-16T15:33:30.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1982 12 1832 1820 0.0573838 0 - 0.957344 2024-04-16T15:33:31.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1983 12 1832 1820 0.0573549 0 - 0.957344 2024-04-16T15:33:32.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1984 12 1832 1820 0.057326 0 - 0.957344 2024-04-16T15:33:33.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1985 12 1832 1820 0.0572971 0 - 0.957344 2024-04-16T15:33:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:33:34.428 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:33:34.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1986 12 1832 1820 0.0572683 0 - 0.957344 2024-04-16T15:33:35.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1987 12 1832 1820 0.0572394 0 - 0.957344 2024-04-16T15:33:36.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:33:36.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:33:36.552+0000 7feeb6a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:33:36.755 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1988 12 1832 1820 0.0572106 0 - 0.957344 2024-04-16T15:33:37.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1989 12 1832 1820 0.0571819 0 - 0.957344 2024-04-16T15:33:38.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1990 12 1832 1820 0.0571531 0 - 0.957344 2024-04-16T15:33:39.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1991 12 1832 1820 0.0571244 0 - 0.957344 2024-04-16T15:33:40.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1992 12 1832 1820 0.0570958 0 - 0.957344 2024-04-16T15:33:41.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1993 12 1832 1820 0.0570671 0 - 0.957344 2024-04-16T15:33:42.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1994 12 1832 1820 0.0570385 0 - 0.957344 2024-04-16T15:33:43.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1995 12 1832 1820 0.0570099 0 - 0.957344 2024-04-16T15:33:44.756 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1996 12 1832 1820 0.0569813 0 - 0.957344 2024-04-16T15:33:45.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1997 12 1832 1820 0.0569528 0 - 0.957344 2024-04-16T15:33:46.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1998 12 1832 1820 0.0569243 0 - 0.957344 2024-04-16T15:33:47.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 1999 12 1832 1820 0.0568958 0 - 0.957344 2024-04-16T15:33:48.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:33:48.759909+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:33:48.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:33:48.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2000 12 1832 1820 0.0568674 0 - 0.957344 2024-04-16T15:33:49.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2001 12 1832 1820 0.056839 0 - 0.957344 2024-04-16T15:33:50.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2002 12 1832 1820 0.0568106 0 - 0.957344 2024-04-16T15:33:51.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2003 12 1832 1820 0.0567822 0 - 0.957344 2024-04-16T15:33:52.757 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2004 12 1832 1820 0.0567539 0 - 0.957344 2024-04-16T15:33:53.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2005 12 1832 1820 0.0567256 0 - 0.957344 2024-04-16T15:33:54.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2006 12 1832 1820 0.0566973 0 - 0.957344 2024-04-16T15:33:55.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2007 12 1832 1820 0.056669 0 - 0.957344 2024-04-16T15:33:56.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2008 12 1832 1820 0.0566408 0 - 0.957344 2024-04-16T15:33:57.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2009 12 1832 1820 0.0566126 0 - 0.957344 2024-04-16T15:33:58.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2010 12 1832 1820 0.0565845 0 - 0.957344 2024-04-16T15:33:59.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2011 12 1832 1820 0.0565563 0 - 0.957344 2024-04-16T15:34:00.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2012 12 1832 1820 0.0565282 0 - 0.957344 2024-04-16T15:34:01.758 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2013 12 1832 1820 0.0565001 0 - 0.957344 2024-04-16T15:34:02.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2014 12 1832 1820 0.0564721 0 - 0.957344 2024-04-16T15:34:03.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2015 12 1832 1820 0.0564441 0 - 0.957344 2024-04-16T15:34:04.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2016 12 1832 1820 0.0564161 0 - 0.957344 2024-04-16T15:34:05.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2017 12 1832 1820 0.0563881 0 - 0.957344 2024-04-16T15:34:06.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2018 12 1832 1820 0.0563601 0 - 0.957344 2024-04-16T15:34:07.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2019 12 1832 1820 0.0563322 0 - 0.957344 2024-04-16T15:34:08.759 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:34:08.762301+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:34:08.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:34:08.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2020 12 1832 1820 0.0563043 0 - 0.957344 2024-04-16T15:34:09.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2021 12 1832 1820 0.0562765 0 - 0.957344 2024-04-16T15:34:10.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2022 12 1832 1820 0.0562487 0 - 0.957344 2024-04-16T15:34:11.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2023 12 1832 1820 0.0562209 0 - 0.957344 2024-04-16T15:34:12.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2024 12 1832 1820 0.0561931 0 - 0.957344 2024-04-16T15:34:13.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2025 12 1832 1820 0.0561653 0 - 0.957344 2024-04-16T15:34:14.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2026 12 1832 1820 0.0561376 0 - 0.957344 2024-04-16T15:34:15.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2027 12 1832 1820 0.0561099 0 - 0.957344 2024-04-16T15:34:16.760 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2028 12 1832 1820 0.0560822 0 - 0.957344 2024-04-16T15:34:17.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2029 12 1832 1820 0.0560546 0 - 0.957344 2024-04-16T15:34:18.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2030 12 1832 1820 0.056027 0 - 0.957344 2024-04-16T15:34:19.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2031 12 1832 1820 0.0559994 0 - 0.957344 2024-04-16T15:34:20.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2032 12 1832 1820 0.0559718 0 - 0.957344 2024-04-16T15:34:21.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2033 12 1832 1820 0.0559443 0 - 0.957344 2024-04-16T15:34:22.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2034 12 1832 1820 0.0559168 0 - 0.957344 2024-04-16T15:34:23.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2035 12 1832 1820 0.0558893 0 - 0.957344 2024-04-16T15:34:24.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2036 12 1832 1820 0.0558619 0 - 0.957344 2024-04-16T15:34:25.761 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2037 12 1832 1820 0.0558345 0 - 0.957344 2024-04-16T15:34:26.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2038 12 1832 1820 0.0558071 0 - 0.957344 2024-04-16T15:34:27.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2039 12 1832 1820 0.0557797 0 - 0.957344 2024-04-16T15:34:28.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:34:28.764765+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:34:28.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:34:28.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2040 12 1832 1820 0.0557524 0 - 0.957344 2024-04-16T15:34:29.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2041 12 1832 1820 0.055725 0 - 0.957344 2024-04-16T15:34:30.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2042 12 1832 1820 0.0556977 0 - 0.957344 2024-04-16T15:34:31.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2043 12 1832 1820 0.0556705 0 - 0.957344 2024-04-16T15:34:32.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2044 12 1832 1820 0.0556432 0 - 0.957344 2024-04-16T15:34:33.762 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2045 12 1832 1820 0.055616 0 - 0.957344 2024-04-16T15:34:34.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2046 12 1832 1820 0.0555889 0 - 0.957344 2024-04-16T15:34:35.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2047 12 1832 1820 0.0555617 0 - 0.957344 2024-04-16T15:34:36.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2048 12 1832 1820 0.0555346 0 - 0.957344 2024-04-16T15:34:37.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2049 12 1832 1820 0.0555075 0 - 0.957344 2024-04-16T15:34:38.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2050 12 1832 1820 0.0554804 0 - 0.957344 2024-04-16T15:34:39.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2051 12 1832 1820 0.0554533 0 - 0.957344 2024-04-16T15:34:40.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2052 12 1832 1820 0.0554263 0 - 0.957344 2024-04-16T15:34:41.763 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2053 12 1832 1820 0.0553993 0 - 0.957344 2024-04-16T15:34:42.764 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2054 12 1832 1820 0.0553723 0 - 0.957344 2024-04-16T15:34:43.764 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2055 12 1832 1820 0.0553454 0 - 0.957344 2024-04-16T15:34:44.764 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2056 12 1832 1820 0.0553185 0 - 0.957344 2024-04-16T15:34:45.764 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2057 12 1832 1820 0.0552916 0 - 0.957344 2024-04-16T15:34:46.764 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2058 12 1832 1820 0.0552647 0 - 0.957344 2024-04-16T15:34:47.764 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2059 12 1832 1820 0.0552379 0 - 0.957344 2024-04-16T15:34:48.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:34:48.767272+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:34:48.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:34:48.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2060 12 1832 1820 0.0552111 0 - 0.957344 2024-04-16T15:34:49.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2061 12 1832 1820 0.0551843 0 - 0.957344 2024-04-16T15:34:50.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2062 12 1832 1820 0.0551575 0 - 0.957344 2024-04-16T15:34:51.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2063 12 1832 1820 0.0551308 0 - 0.957344 2024-04-16T15:34:52.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2064 12 1832 1820 0.0551041 0 - 0.957344 2024-04-16T15:34:53.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2065 12 1832 1820 0.0550774 0 - 0.957344 2024-04-16T15:34:54.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2066 12 1832 1820 0.0550507 0 - 0.957344 2024-04-16T15:34:55.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2067 12 1832 1820 0.0550241 0 - 0.957344 2024-04-16T15:34:56.765 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2068 12 1832 1820 0.0549975 0 - 0.957344 2024-04-16T15:34:57.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2069 12 1832 1820 0.0549709 0 - 0.957344 2024-04-16T15:34:58.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2070 12 1832 1820 0.0549443 0 - 0.957344 2024-04-16T15:34:59.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2071 12 1832 1820 0.0549178 0 - 0.957344 2024-04-16T15:35:00.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2072 12 1832 1820 0.0548913 0 - 0.957344 2024-04-16T15:35:01.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2073 12 1832 1820 0.0548648 0 - 0.957344 2024-04-16T15:35:02.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2074 12 1832 1820 0.0548384 0 - 0.957344 2024-04-16T15:35:03.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2075 12 1832 1820 0.054812 0 - 0.957344 2024-04-16T15:35:04.766 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2076 12 1832 1820 0.0547856 0 - 0.957344 2024-04-16T15:35:05.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2077 12 1832 1820 0.0547592 0 - 0.957344 2024-04-16T15:35:06.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2078 12 1832 1820 0.0547328 0 - 0.957344 2024-04-16T15:35:07.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2079 12 1832 1820 0.0547065 0 - 0.957344 2024-04-16T15:35:08.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:35:08.769983+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:35:08.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:35:08.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2080 12 1832 1820 0.0546802 0 - 0.957344 2024-04-16T15:35:09.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2081 12 1832 1820 0.0546539 0 - 0.957344 2024-04-16T15:35:10.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2082 12 1832 1820 0.0546277 0 - 0.957344 2024-04-16T15:35:11.767 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2083 12 1832 1820 0.0546014 0 - 0.957344 2024-04-16T15:35:12.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2084 12 1832 1820 0.0545752 0 - 0.957344 2024-04-16T15:35:13.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2085 12 1832 1820 0.0545491 0 - 0.957344 2024-04-16T15:35:14.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2086 12 1832 1820 0.0545229 0 - 0.957344 2024-04-16T15:35:15.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2087 12 1832 1820 0.0544968 0 - 0.957344 2024-04-16T15:35:16.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2088 12 1832 1820 0.0544707 0 - 0.957344 2024-04-16T15:35:17.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2089 12 1832 1820 0.0544446 0 - 0.957344 2024-04-16T15:35:18.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2090 12 1832 1820 0.0544186 0 - 0.957344 2024-04-16T15:35:19.768 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2091 12 1832 1820 0.0543925 0 - 0.957344 2024-04-16T15:35:20.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2092 12 1832 1820 0.0543665 0 - 0.957344 2024-04-16T15:35:21.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2093 12 1832 1820 0.0543406 0 - 0.957344 2024-04-16T15:35:22.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2094 12 1832 1820 0.0543146 0 - 0.957344 2024-04-16T15:35:23.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2095 12 1832 1820 0.0542887 0 - 0.957344 2024-04-16T15:35:24.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2096 12 1832 1820 0.0542628 0 - 0.957344 2024-04-16T15:35:25.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2097 12 1832 1820 0.0542369 0 - 0.957344 2024-04-16T15:35:26.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2098 12 1832 1820 0.0542111 0 - 0.957344 2024-04-16T15:35:27.769 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2099 12 1832 1820 0.0541852 0 - 0.957344 2024-04-16T15:35:28.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:35:28.772593+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:35:28.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:35:28.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2100 12 1832 1820 0.0541594 0 - 0.957344 2024-04-16T15:35:29.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2101 12 1832 1820 0.0541337 0 - 0.957344 2024-04-16T15:35:30.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2102 12 1832 1820 0.0541079 0 - 0.957344 2024-04-16T15:35:31.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2103 12 1832 1820 0.0540822 0 - 0.957344 2024-04-16T15:35:32.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2104 12 1832 1820 0.0540565 0 - 0.957344 2024-04-16T15:35:33.770 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2105 12 1832 1820 0.0540308 0 - 0.957344 2024-04-16T15:35:34.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2106 12 1832 1820 0.0540051 0 - 0.957344 2024-04-16T15:35:35.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2107 12 1832 1820 0.0539795 0 - 0.957344 2024-04-16T15:35:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:35:36.470 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:35:36.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2108 12 1832 1820 0.0539539 0 - 0.957344 2024-04-16T15:35:37.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2109 12 1832 1820 0.0539283 0 - 0.957344 2024-04-16T15:35:38.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:35:38.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:35:38.594+0000 7f2062d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:35:38.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2110 12 1832 1820 0.0539027 0 - 0.957344 2024-04-16T15:35:39.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2111 12 1832 1820 0.0538772 0 - 0.957344 2024-04-16T15:35:40.771 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2112 12 1832 1820 0.0538517 0 - 0.957344 2024-04-16T15:35:41.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2113 12 1832 1820 0.0538262 0 - 0.957344 2024-04-16T15:35:42.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2114 12 1832 1820 0.0538008 0 - 0.957344 2024-04-16T15:35:43.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2115 12 1832 1820 0.0537753 0 - 0.957344 2024-04-16T15:35:44.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2116 12 1832 1820 0.0537499 0 - 0.957344 2024-04-16T15:35:45.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2117 12 1832 1820 0.0537245 0 - 0.957344 2024-04-16T15:35:46.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2118 12 1832 1820 0.0536991 0 - 0.957344 2024-04-16T15:35:47.772 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2119 12 1832 1820 0.0536738 0 - 0.957344 2024-04-16T15:35:48.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:35:48.775462+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:35:48.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:35:48.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2120 12 1832 1820 0.0536485 0 - 0.957344 2024-04-16T15:35:49.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2121 12 1832 1820 0.0536232 0 - 0.957344 2024-04-16T15:35:50.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2122 12 1832 1820 0.0535979 0 - 0.957344 2024-04-16T15:35:51.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2123 12 1832 1820 0.0535727 0 - 0.957344 2024-04-16T15:35:52.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2124 12 1832 1820 0.0535475 0 - 0.957344 2024-04-16T15:35:53.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2125 12 1832 1820 0.0535223 0 - 0.957344 2024-04-16T15:35:54.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2126 12 1832 1820 0.0534971 0 - 0.957344 2024-04-16T15:35:55.773 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2127 12 1832 1820 0.0534719 0 - 0.957344 2024-04-16T15:35:56.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2128 12 1832 1820 0.0534468 0 - 0.957344 2024-04-16T15:35:57.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2129 12 1832 1820 0.0534217 0 - 0.957344 2024-04-16T15:35:58.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2130 12 1832 1820 0.0533966 0 - 0.957344 2024-04-16T15:35:59.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2131 12 1832 1820 0.0533716 0 - 0.957344 2024-04-16T15:36:00.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2132 12 1832 1820 0.0533465 0 - 0.957344 2024-04-16T15:36:01.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2133 12 1832 1820 0.0533215 0 - 0.957344 2024-04-16T15:36:02.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2134 12 1832 1820 0.0532965 0 - 0.957344 2024-04-16T15:36:03.774 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2135 12 1832 1820 0.0532716 0 - 0.957344 2024-04-16T15:36:04.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2136 12 1832 1820 0.0532466 0 - 0.957344 2024-04-16T15:36:05.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2137 12 1832 1820 0.0532217 0 - 0.957344 2024-04-16T15:36:06.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2138 12 1832 1820 0.0531968 0 - 0.957344 2024-04-16T15:36:07.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2139 12 1832 1820 0.0531719 0 - 0.957344 2024-04-16T15:36:08.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:36:08.778153+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:36:08.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:36:08.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2140 12 1832 1820 0.0531471 0 - 0.957344 2024-04-16T15:36:09.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2141 12 1832 1820 0.0531223 0 - 0.957344 2024-04-16T15:36:10.775 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2142 12 1832 1820 0.0530975 0 - 0.957344 2024-04-16T15:36:11.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2143 12 1832 1820 0.0530727 0 - 0.957344 2024-04-16T15:36:12.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2144 12 1832 1820 0.0530479 0 - 0.957344 2024-04-16T15:36:13.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2145 12 1832 1820 0.0530232 0 - 0.957344 2024-04-16T15:36:14.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2146 12 1832 1820 0.0529985 0 - 0.957344 2024-04-16T15:36:15.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2147 12 1832 1820 0.0529738 0 - 0.957344 2024-04-16T15:36:16.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2148 12 1832 1820 0.0529492 0 - 0.957344 2024-04-16T15:36:17.776 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2149 12 1832 1820 0.0529245 0 - 0.957344 2024-04-16T15:36:18.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2150 12 1832 1820 0.0528999 0 - 0.957344 2024-04-16T15:36:19.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2151 12 1832 1820 0.0528753 0 - 0.957344 2024-04-16T15:36:20.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2152 12 1832 1820 0.0528507 0 - 0.957344 2024-04-16T15:36:21.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2153 12 1832 1820 0.0528262 0 - 0.957344 2024-04-16T15:36:22.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2154 12 1832 1820 0.0528017 0 - 0.957344 2024-04-16T15:36:23.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2155 12 1832 1820 0.0527772 0 - 0.957344 2024-04-16T15:36:24.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2156 12 1832 1820 0.0527527 0 - 0.957344 2024-04-16T15:36:25.777 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2157 12 1832 1820 0.0527282 0 - 0.957344 2024-04-16T15:36:26.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2158 12 1832 1820 0.0527038 0 - 0.957344 2024-04-16T15:36:27.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2159 12 1832 1820 0.0526794 0 - 0.957344 2024-04-16T15:36:28.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:36:28.780906+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:36:28.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:36:28.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2160 12 1832 1820 0.052655 0 - 0.957344 2024-04-16T15:36:29.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2161 12 1832 1820 0.0526306 0 - 0.957344 2024-04-16T15:36:30.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2162 12 1832 1820 0.0526063 0 - 0.957344 2024-04-16T15:36:31.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2163 12 1832 1820 0.052582 0 - 0.957344 2024-04-16T15:36:32.778 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2164 12 1832 1820 0.0525577 0 - 0.957344 2024-04-16T15:36:33.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2165 12 1832 1820 0.0525334 0 - 0.957344 2024-04-16T15:36:34.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2166 12 1832 1820 0.0525091 0 - 0.957344 2024-04-16T15:36:35.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2167 12 1832 1820 0.0524849 0 - 0.957344 2024-04-16T15:36:36.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2168 12 1832 1820 0.0524607 0 - 0.957344 2024-04-16T15:36:37.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2169 12 1832 1820 0.0524365 0 - 0.957344 2024-04-16T15:36:38.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2170 12 1832 1820 0.0524123 0 - 0.957344 2024-04-16T15:36:39.779 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2171 12 1832 1820 0.0523882 0 - 0.957344 2024-04-16T15:36:40.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2172 12 1832 1820 0.0523641 0 - 0.957344 2024-04-16T15:36:41.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2173 12 1832 1820 0.05234 0 - 0.957344 2024-04-16T15:36:42.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2174 12 1832 1820 0.0523159 0 - 0.957344 2024-04-16T15:36:43.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2175 12 1832 1820 0.0522919 0 - 0.957344 2024-04-16T15:36:44.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2176 12 1832 1820 0.0522678 0 - 0.957344 2024-04-16T15:36:45.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2177 12 1832 1820 0.0522438 0 - 0.957344 2024-04-16T15:36:46.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2178 12 1832 1820 0.0522198 0 - 0.957344 2024-04-16T15:36:47.780 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2179 12 1832 1820 0.0521959 0 - 0.957344 2024-04-16T15:36:48.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:36:48.783538+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:36:48.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:36:48.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2180 12 1832 1820 0.0521719 0 - 0.957344 2024-04-16T15:36:49.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2181 12 1832 1820 0.052148 0 - 0.957344 2024-04-16T15:36:50.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2182 12 1832 1820 0.0521241 0 - 0.957344 2024-04-16T15:36:51.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2183 12 1832 1820 0.0521002 0 - 0.957344 2024-04-16T15:36:52.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2184 12 1832 1820 0.0520764 0 - 0.957344 2024-04-16T15:36:53.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2185 12 1832 1820 0.0520525 0 - 0.957344 2024-04-16T15:36:54.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2186 12 1832 1820 0.0520287 0 - 0.957344 2024-04-16T15:36:55.781 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2187 12 1832 1820 0.0520049 0 - 0.957344 2024-04-16T15:36:56.782 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2188 12 1832 1820 0.0519812 0 - 0.957344 2024-04-16T15:36:57.782 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2189 12 1832 1820 0.0519574 0 - 0.957344 2024-04-16T15:36:58.782 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2190 12 1832 1820 0.0519337 0 - 0.957344 2024-04-16T15:36:59.782 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2191 12 1832 1820 0.05191 0 - 0.957344 2024-04-16T15:37:00.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2192 12 1832 1820 0.0518863 0 - 0.957344 2024-04-16T15:37:01.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2193 12 1832 1820 0.0518627 0 - 0.957344 2024-04-16T15:37:02.782 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2194 12 1832 1820 0.051839 0 - 0.957344 2024-04-16T15:37:03.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2195 12 1832 1820 0.0518154 0 - 0.957344 2024-04-16T15:37:04.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2196 12 1832 1820 0.0517918 0 - 0.957344 2024-04-16T15:37:05.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2197 12 1832 1820 0.0517682 0 - 0.957344 2024-04-16T15:37:06.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2198 12 1832 1820 0.0517447 0 - 0.957344 2024-04-16T15:37:07.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2199 12 1832 1820 0.0517211 0 - 0.957344 2024-04-16T15:37:08.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:37:08.786137+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:37:08.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:37:08.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2200 12 1832 1820 0.0516976 0 - 0.957344 2024-04-16T15:37:09.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2201 12 1832 1820 0.0516741 0 - 0.957344 2024-04-16T15:37:10.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2202 12 1832 1820 0.0516507 0 - 0.957344 2024-04-16T15:37:11.783 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2203 12 1832 1820 0.0516272 0 - 0.957344 2024-04-16T15:37:12.784 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2204 12 1832 1820 0.0516038 0 - 0.957344 2024-04-16T15:37:13.784 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2205 12 1832 1820 0.0515804 0 - 0.957344 2024-04-16T15:37:14.784 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2206 12 1832 1820 0.051557 0 - 0.957344 2024-04-16T15:37:15.784 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2207 12 1832 1820 0.0515337 0 - 0.957344 2024-04-16T15:37:16.784 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2208 12 1832 1820 0.0515103 0 - 0.957344 2024-04-16T15:37:17.784 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2209 12 1832 1820 0.051487 0 - 0.957344 2024-04-16T15:37:18.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2210 12 1832 1820 0.0514637 0 - 0.957344 2024-04-16T15:37:19.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2211 12 1832 1820 0.0514404 0 - 0.957344 2024-04-16T15:37:20.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2212 12 1832 1820 0.0514172 0 - 0.957344 2024-04-16T15:37:21.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2213 12 1832 1820 0.0513939 0 - 0.957344 2024-04-16T15:37:22.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2214 12 1832 1820 0.0513707 0 - 0.957344 2024-04-16T15:37:23.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2215 12 1832 1820 0.0513475 0 - 0.957344 2024-04-16T15:37:24.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2216 12 1832 1820 0.0513244 0 - 0.957344 2024-04-16T15:37:25.785 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2217 12 1832 1820 0.0513012 0 - 0.957344 2024-04-16T15:37:26.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2218 12 1832 1820 0.0512781 0 - 0.957344 2024-04-16T15:37:27.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2219 12 1832 1820 0.051255 0 - 0.957344 2024-04-16T15:37:28.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:37:28.788718+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:37:28.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:37:28.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2220 12 1832 1820 0.0512319 0 - 0.957344 2024-04-16T15:37:29.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2221 12 1832 1820 0.0512088 0 - 0.957344 2024-04-16T15:37:30.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2222 12 1832 1820 0.0511858 0 - 0.957344 2024-04-16T15:37:31.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2223 12 1832 1820 0.0511628 0 - 0.957344 2024-04-16T15:37:32.786 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2224 12 1832 1820 0.0511397 0 - 0.957344 2024-04-16T15:37:33.787 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2225 12 1832 1820 0.0511168 0 - 0.957344 2024-04-16T15:37:34.787 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2226 12 1832 1820 0.0510938 0 - 0.957344 2024-04-16T15:37:35.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2227 12 1832 1820 0.0510709 0 - 0.957344 2024-04-16T15:37:36.787 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2228 12 1832 1820 0.0510479 0 - 0.957344 2024-04-16T15:37:37.787 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2229 12 1832 1820 0.051025 0 - 0.957344 2024-04-16T15:37:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:37:38.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:37:38.787 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2230 12 1832 1820 0.0510022 0 - 0.957344 2024-04-16T15:37:39.787 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2231 12 1832 1820 0.0509793 0 - 0.957344 2024-04-16T15:37:40.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:37:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:37:40.635+0000 7f2d08ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:37:40.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2232 12 1832 1820 0.0509565 0 - 0.957344 2024-04-16T15:37:41.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2233 12 1832 1820 0.0509336 0 - 0.957344 2024-04-16T15:37:42.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2234 12 1832 1820 0.0509108 0 - 0.957344 2024-04-16T15:37:43.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2235 12 1832 1820 0.0508881 0 - 0.957344 2024-04-16T15:37:44.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2236 12 1832 1820 0.0508653 0 - 0.957344 2024-04-16T15:37:45.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2237 12 1832 1820 0.0508426 0 - 0.957344 2024-04-16T15:37:46.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2238 12 1832 1820 0.0508198 0 - 0.957344 2024-04-16T15:37:47.788 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2239 12 1832 1820 0.0507971 0 - 0.957344 2024-04-16T15:37:48.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:37:48.791557+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:37:48.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:37:48.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2240 12 1832 1820 0.0507745 0 - 0.957344 2024-04-16T15:37:49.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2241 12 1832 1820 0.0507518 0 - 0.957344 2024-04-16T15:37:50.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2242 12 1832 1820 0.0507292 0 - 0.957344 2024-04-16T15:37:51.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2243 12 1832 1820 0.0507066 0 - 0.957344 2024-04-16T15:37:52.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2244 12 1832 1820 0.050684 0 - 0.957344 2024-04-16T15:37:53.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2245 12 1832 1820 0.0506614 0 - 0.957344 2024-04-16T15:37:54.789 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2246 12 1832 1820 0.0506388 0 - 0.957344 2024-04-16T15:37:55.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2247 12 1832 1820 0.0506163 0 - 0.957344 2024-04-16T15:37:56.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2248 12 1832 1820 0.0505938 0 - 0.957344 2024-04-16T15:37:57.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2249 12 1832 1820 0.0505713 0 - 0.957344 2024-04-16T15:37:58.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2250 12 1832 1820 0.0505488 0 - 0.957344 2024-04-16T15:37:59.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2251 12 1832 1820 0.0505263 0 - 0.957344 2024-04-16T15:38:00.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2252 12 1832 1820 0.0505039 0 - 0.957344 2024-04-16T15:38:01.790 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2253 12 1832 1820 0.0504815 0 - 0.957344 2024-04-16T15:38:02.791 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2254 12 1832 1820 0.0504591 0 - 0.957344 2024-04-16T15:38:03.791 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2255 12 1832 1820 0.0504367 0 - 0.957344 2024-04-16T15:38:04.791 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2256 12 1832 1820 0.0504144 0 - 0.957344 2024-04-16T15:38:05.791 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2257 12 1832 1820 0.050392 0 - 0.957344 2024-04-16T15:38:06.791 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2258 12 1832 1820 0.0503697 0 - 0.957344 2024-04-16T15:38:07.791 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2259 12 1832 1820 0.0503474 0 - 0.957344 2024-04-16T15:38:08.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:38:08.794255+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:38:08.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:38:08.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2260 12 1832 1820 0.0503251 0 - 0.957344 2024-04-16T15:38:09.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2261 12 1832 1820 0.0503029 0 - 0.957344 2024-04-16T15:38:10.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2262 12 1832 1820 0.0502806 0 - 0.957344 2024-04-16T15:38:11.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2263 12 1832 1820 0.0502584 0 - 0.957344 2024-04-16T15:38:12.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2264 12 1832 1820 0.0502362 0 - 0.957344 2024-04-16T15:38:13.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2265 12 1832 1820 0.050214 0 - 0.957344 2024-04-16T15:38:14.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2266 12 1832 1820 0.0501919 0 - 0.957344 2024-04-16T15:38:15.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2267 12 1832 1820 0.0501697 0 - 0.957344 2024-04-16T15:38:16.792 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2268 12 1832 1820 0.0501476 0 - 0.957344 2024-04-16T15:38:17.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2269 12 1832 1820 0.0501255 0 - 0.957344 2024-04-16T15:38:18.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2270 12 1832 1820 0.0501034 0 - 0.957344 2024-04-16T15:38:19.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2271 12 1832 1820 0.0500814 0 - 0.957344 2024-04-16T15:38:20.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2272 12 1832 1820 0.0500593 0 - 0.957344 2024-04-16T15:38:21.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2273 12 1832 1820 0.0500373 0 - 0.957344 2024-04-16T15:38:22.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2274 12 1832 1820 0.0500153 0 - 0.957344 2024-04-16T15:38:23.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2275 12 1832 1820 0.0499933 0 - 0.957344 2024-04-16T15:38:24.793 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2276 12 1832 1820 0.0499714 0 - 0.957344 2024-04-16T15:38:25.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2277 12 1832 1820 0.0499494 0 - 0.957344 2024-04-16T15:38:26.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2278 12 1832 1820 0.0499275 0 - 0.957344 2024-04-16T15:38:27.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2279 12 1832 1820 0.0499056 0 - 0.957344 2024-04-16T15:38:28.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:38:28.796853+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:38:28.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:38:28.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2280 12 1832 1820 0.0498837 0 - 0.957344 2024-04-16T15:38:29.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2281 12 1832 1820 0.0498618 0 - 0.957344 2024-04-16T15:38:30.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2282 12 1832 1820 0.04984 0 - 0.957344 2024-04-16T15:38:31.794 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2283 12 1832 1820 0.0498181 0 - 0.957344 2024-04-16T15:38:32.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2284 12 1832 1820 0.0497963 0 - 0.957344 2024-04-16T15:38:33.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2285 12 1832 1820 0.0497745 0 - 0.957344 2024-04-16T15:38:34.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2286 12 1832 1820 0.0497528 0 - 0.957344 2024-04-16T15:38:35.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2287 12 1832 1820 0.049731 0 - 0.957344 2024-04-16T15:38:36.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2288 12 1832 1820 0.0497093 0 - 0.957344 2024-04-16T15:38:37.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2289 12 1832 1820 0.0496875 0 - 0.957344 2024-04-16T15:38:38.795 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2290 12 1832 1820 0.0496658 0 - 0.957344 2024-04-16T15:38:39.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2291 12 1832 1820 0.0496442 0 - 0.957344 2024-04-16T15:38:40.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2292 12 1832 1820 0.0496225 0 - 0.957344 2024-04-16T15:38:41.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2293 12 1832 1820 0.0496009 0 - 0.957344 2024-04-16T15:38:42.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2294 12 1832 1820 0.0495792 0 - 0.957344 2024-04-16T15:38:43.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2295 12 1832 1820 0.0495576 0 - 0.957344 2024-04-16T15:38:44.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2296 12 1832 1820 0.0495361 0 - 0.957344 2024-04-16T15:38:45.796 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2297 12 1832 1820 0.0495145 0 - 0.957344 2024-04-16T15:38:46.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2298 12 1832 1820 0.0494929 0 - 0.957344 2024-04-16T15:38:47.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2299 12 1832 1820 0.0494714 0 - 0.957344 2024-04-16T15:38:48.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:38:48.799674+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:38:48.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:38:48.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2300 12 1832 1820 0.0494499 0 - 0.957344 2024-04-16T15:38:49.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2301 12 1832 1820 0.0494284 0 - 0.957344 2024-04-16T15:38:50.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2302 12 1832 1820 0.0494069 0 - 0.957344 2024-04-16T15:38:51.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2303 12 1832 1820 0.0493855 0 - 0.957344 2024-04-16T15:38:52.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2304 12 1832 1820 0.0493641 0 - 0.957344 2024-04-16T15:38:53.797 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2305 12 1832 1820 0.0493426 0 - 0.957344 2024-04-16T15:38:54.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2306 12 1832 1820 0.0493212 0 - 0.957344 2024-04-16T15:38:55.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2307 12 1832 1820 0.0492999 0 - 0.957344 2024-04-16T15:38:56.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2308 12 1832 1820 0.0492785 0 - 0.957344 2024-04-16T15:38:57.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2309 12 1832 1820 0.0492572 0 - 0.957344 2024-04-16T15:38:58.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2310 12 1832 1820 0.0492358 0 - 0.957344 2024-04-16T15:38:59.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2311 12 1832 1820 0.0492145 0 - 0.957344 2024-04-16T15:39:00.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2312 12 1832 1820 0.0491932 0 - 0.957344 2024-04-16T15:39:01.798 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2313 12 1832 1820 0.049172 0 - 0.957344 2024-04-16T15:39:02.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2314 12 1832 1820 0.0491507 0 - 0.957344 2024-04-16T15:39:03.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2315 12 1832 1820 0.0491295 0 - 0.957344 2024-04-16T15:39:04.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2316 12 1832 1820 0.0491083 0 - 0.957344 2024-04-16T15:39:05.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2317 12 1832 1820 0.0490871 0 - 0.957344 2024-04-16T15:39:06.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2318 12 1832 1820 0.0490659 0 - 0.957344 2024-04-16T15:39:07.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2319 12 1832 1820 0.0490448 0 - 0.957344 2024-04-16T15:39:08.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:39:08.802083+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:39:08.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:39:08.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2320 12 1832 1820 0.0490236 0 - 0.957344 2024-04-16T15:39:09.799 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2321 12 1832 1820 0.0490025 0 - 0.957344 2024-04-16T15:39:10.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2322 12 1832 1820 0.0489814 0 - 0.957344 2024-04-16T15:39:11.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2323 12 1832 1820 0.0489603 0 - 0.957344 2024-04-16T15:39:12.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2324 12 1832 1820 0.0489392 0 - 0.957344 2024-04-16T15:39:13.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2325 12 1832 1820 0.0489182 0 - 0.957344 2024-04-16T15:39:14.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2326 12 1832 1820 0.0488972 0 - 0.957344 2024-04-16T15:39:15.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2327 12 1832 1820 0.0488762 0 - 0.957344 2024-04-16T15:39:16.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2328 12 1832 1820 0.0488552 0 - 0.957344 2024-04-16T15:39:17.800 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2329 12 1832 1820 0.0488342 0 - 0.957344 2024-04-16T15:39:18.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2330 12 1832 1820 0.0488132 0 - 0.957344 2024-04-16T15:39:19.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2331 12 1832 1820 0.0487923 0 - 0.957344 2024-04-16T15:39:20.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2332 12 1832 1820 0.0487714 0 - 0.957344 2024-04-16T15:39:21.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2333 12 1832 1820 0.0487504 0 - 0.957344 2024-04-16T15:39:22.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2334 12 1832 1820 0.0487296 0 - 0.957344 2024-04-16T15:39:23.801 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2335 12 1832 1820 0.0487087 0 - 0.957344 2024-04-16T15:39:24.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2336 12 1832 1820 0.0486878 0 - 0.957344 2024-04-16T15:39:25.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2337 12 1832 1820 0.048667 0 - 0.957344 2024-04-16T15:39:26.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2338 12 1832 1820 0.0486462 0 - 0.957344 2024-04-16T15:39:27.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2339 12 1832 1820 0.0486254 0 - 0.957344 2024-04-16T15:39:28.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:39:28.804817+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:39:28.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:39:28.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2340 12 1832 1820 0.0486046 0 - 0.957344 2024-04-16T15:39:29.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2341 12 1832 1820 0.0485839 0 - 0.957344 2024-04-16T15:39:30.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2342 12 1832 1820 0.0485631 0 - 0.957344 2024-04-16T15:39:31.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2343 12 1832 1820 0.0485424 0 - 0.957344 2024-04-16T15:39:32.802 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2344 12 1832 1820 0.0485217 0 - 0.957344 2024-04-16T15:39:33.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2345 12 1832 1820 0.048501 0 - 0.957344 2024-04-16T15:39:34.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2346 12 1832 1820 0.0484803 0 - 0.957344 2024-04-16T15:39:35.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2347 12 1832 1820 0.0484596 0 - 0.957344 2024-04-16T15:39:36.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2348 12 1832 1820 0.048439 0 - 0.957344 2024-04-16T15:39:37.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2349 12 1832 1820 0.0484184 0 - 0.957344 2024-04-16T15:39:38.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2350 12 1832 1820 0.0483978 0 - 0.957344 2024-04-16T15:39:39.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2351 12 1832 1820 0.0483772 0 - 0.957344 2024-04-16T15:39:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:39:40.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:39:40.803 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2352 12 1832 1820 0.0483566 0 - 0.957344 2024-04-16T15:39:41.804 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2353 12 1832 1820 0.0483361 0 - 0.957344 2024-04-16T15:39:42.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:39:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:39:42.678+0000 7fc73166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:39:42.804 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2354 12 1832 1820 0.0483155 0 - 0.957344 2024-04-16T15:39:43.804 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2355 12 1832 1820 0.048295 0 - 0.957344 2024-04-16T15:39:44.804 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2356 12 1832 1820 0.0482745 0 - 0.957344 2024-04-16T15:39:45.804 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2357 12 1832 1820 0.0482541 0 - 0.957344 2024-04-16T15:39:46.804 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2358 12 1832 1820 0.0482336 0 - 0.957344 2024-04-16T15:39:47.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2359 12 1832 1820 0.0482131 0 - 0.957344 2024-04-16T15:39:48.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:39:48.807515+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:39:48.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:39:48.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2360 12 1832 1820 0.0481927 0 - 0.957344 2024-04-16T15:39:49.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2361 12 1832 1820 0.0481723 0 - 0.957344 2024-04-16T15:39:50.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2362 12 1832 1820 0.0481519 0 - 0.957344 2024-04-16T15:39:51.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2363 12 1832 1820 0.0481315 0 - 0.957344 2024-04-16T15:39:52.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2364 12 1832 1820 0.0481112 0 - 0.957344 2024-04-16T15:39:53.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2365 12 1832 1820 0.0480908 0 - 0.957344 2024-04-16T15:39:54.805 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2366 12 1832 1820 0.0480705 0 - 0.957344 2024-04-16T15:39:55.806 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2367 12 1832 1820 0.0480502 0 - 0.957344 2024-04-16T15:39:56.806 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2368 12 1832 1820 0.0480299 0 - 0.957344 2024-04-16T15:39:57.806 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2369 12 1832 1820 0.0480096 0 - 0.957344 2024-04-16T15:39:58.806 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2370 12 1832 1820 0.0479894 0 - 0.957344 2024-04-16T15:39:59.806 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2371 12 1832 1820 0.0479691 0 - 0.957344 2024-04-16T15:40:00.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2372 12 1832 1820 0.0479489 0 - 0.957344 2024-04-16T15:40:01.806 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2373 12 1832 1820 0.0479287 0 - 0.957344 2024-04-16T15:40:02.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2374 12 1832 1820 0.0479085 0 - 0.957344 2024-04-16T15:40:03.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2375 12 1832 1820 0.0478883 0 - 0.957344 2024-04-16T15:40:04.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2376 12 1832 1820 0.0478682 0 - 0.957344 2024-04-16T15:40:05.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2377 12 1832 1820 0.047848 0 - 0.957344 2024-04-16T15:40:06.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2378 12 1832 1820 0.0478279 0 - 0.957344 2024-04-16T15:40:07.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2379 12 1832 1820 0.0478078 0 - 0.957344 2024-04-16T15:40:08.807 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:40:08.810079+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:40:08.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:40:08.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2380 12 1832 1820 0.0477877 0 - 0.957344 2024-04-16T15:40:09.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2381 12 1832 1820 0.0477677 0 - 0.957344 2024-04-16T15:40:10.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2382 12 1832 1820 0.0477476 0 - 0.957344 2024-04-16T15:40:11.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2383 12 1832 1820 0.0477276 0 - 0.957344 2024-04-16T15:40:12.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2384 12 1832 1820 0.0477076 0 - 0.957344 2024-04-16T15:40:13.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2385 12 1832 1820 0.0476875 0 - 0.957344 2024-04-16T15:40:14.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2386 12 1832 1820 0.0476676 0 - 0.957344 2024-04-16T15:40:15.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2387 12 1832 1820 0.0476476 0 - 0.957344 2024-04-16T15:40:16.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2388 12 1832 1820 0.0476276 0 - 0.957344 2024-04-16T15:40:17.808 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2389 12 1832 1820 0.0476077 0 - 0.957344 2024-04-16T15:40:18.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2390 12 1832 1820 0.0475878 0 - 0.957344 2024-04-16T15:40:19.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2391 12 1832 1820 0.0475679 0 - 0.957344 2024-04-16T15:40:20.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2392 12 1832 1820 0.047548 0 - 0.957344 2024-04-16T15:40:21.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2393 12 1832 1820 0.0475281 0 - 0.957344 2024-04-16T15:40:22.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2394 12 1832 1820 0.0475083 0 - 0.957344 2024-04-16T15:40:23.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2395 12 1832 1820 0.0474884 0 - 0.957344 2024-04-16T15:40:24.809 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2396 12 1832 1820 0.0474686 0 - 0.957344 2024-04-16T15:40:25.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2397 12 1832 1820 0.0474488 0 - 0.957344 2024-04-16T15:40:26.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2398 12 1832 1820 0.047429 0 - 0.957344 2024-04-16T15:40:27.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2399 12 1832 1820 0.0474093 0 - 0.957344 2024-04-16T15:40:28.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:40:28.812761+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:40:28.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:40:28.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2400 12 1832 1820 0.0473895 0 - 0.957344 2024-04-16T15:40:29.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2401 12 1832 1820 0.0473698 0 - 0.957344 2024-04-16T15:40:30.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2402 12 1832 1820 0.04735 0 - 0.957344 2024-04-16T15:40:31.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2403 12 1832 1820 0.0473303 0 - 0.957344 2024-04-16T15:40:32.810 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2404 12 1832 1820 0.0473107 0 - 0.957344 2024-04-16T15:40:33.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2405 12 1832 1820 0.047291 0 - 0.957344 2024-04-16T15:40:34.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2406 12 1832 1820 0.0472713 0 - 0.957344 2024-04-16T15:40:35.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2407 12 1832 1820 0.0472517 0 - 0.957344 2024-04-16T15:40:36.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2408 12 1832 1820 0.0472321 0 - 0.957344 2024-04-16T15:40:37.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2409 12 1832 1820 0.0472125 0 - 0.957344 2024-04-16T15:40:38.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2410 12 1832 1820 0.0471929 0 - 0.957344 2024-04-16T15:40:39.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2411 12 1832 1820 0.0471733 0 - 0.957344 2024-04-16T15:40:40.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2412 12 1832 1820 0.0471537 0 - 0.957344 2024-04-16T15:40:41.811 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2413 12 1832 1820 0.0471342 0 - 0.957344 2024-04-16T15:40:42.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2414 12 1832 1820 0.0471147 0 - 0.957344 2024-04-16T15:40:43.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2415 12 1832 1820 0.0470952 0 - 0.957344 2024-04-16T15:40:44.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2416 12 1832 1820 0.0470757 0 - 0.957344 2024-04-16T15:40:45.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2417 12 1832 1820 0.0470562 0 - 0.957344 2024-04-16T15:40:46.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2418 12 1832 1820 0.0470367 0 - 0.957344 2024-04-16T15:40:47.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2419 12 1832 1820 0.0470173 0 - 0.957344 2024-04-16T15:40:48.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:40:48.815110+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:40:48.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:40:48.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2420 12 1832 1820 0.0469979 0 - 0.957344 2024-04-16T15:40:49.812 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2421 12 1832 1820 0.0469784 0 - 0.957344 2024-04-16T15:40:50.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2422 12 1832 1820 0.046959 0 - 0.957344 2024-04-16T15:40:51.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2423 12 1832 1820 0.0469397 0 - 0.957344 2024-04-16T15:40:52.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2424 12 1832 1820 0.0469203 0 - 0.957344 2024-04-16T15:40:53.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2425 12 1832 1820 0.046901 0 - 0.957344 2024-04-16T15:40:54.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2426 12 1832 1820 0.0468816 0 - 0.957344 2024-04-16T15:40:55.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2427 12 1832 1820 0.0468623 0 - 0.957344 2024-04-16T15:40:56.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2428 12 1832 1820 0.046843 0 - 0.957344 2024-04-16T15:40:57.813 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2429 12 1832 1820 0.0468237 0 - 0.957344 2024-04-16T15:40:58.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2430 12 1832 1820 0.0468045 0 - 0.957344 2024-04-16T15:40:59.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2431 12 1832 1820 0.0467852 0 - 0.957344 2024-04-16T15:41:00.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2432 12 1832 1820 0.046766 0 - 0.957344 2024-04-16T15:41:01.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2433 12 1832 1820 0.0467467 0 - 0.957344 2024-04-16T15:41:02.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2434 12 1832 1820 0.0467275 0 - 0.957344 2024-04-16T15:41:03.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2435 12 1832 1820 0.0467083 0 - 0.957344 2024-04-16T15:41:04.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2436 12 1832 1820 0.0466892 0 - 0.957344 2024-04-16T15:41:05.814 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2437 12 1832 1820 0.04667 0 - 0.957344 2024-04-16T15:41:06.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2438 12 1832 1820 0.0466509 0 - 0.957344 2024-04-16T15:41:07.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2439 12 1832 1820 0.0466317 0 - 0.957344 2024-04-16T15:41:08.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:41:08.817754+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:41:08.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:41:08.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2440 12 1832 1820 0.0466126 0 - 0.957344 2024-04-16T15:41:09.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2441 12 1832 1820 0.0465935 0 - 0.957344 2024-04-16T15:41:10.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2442 12 1832 1820 0.0465745 0 - 0.957344 2024-04-16T15:41:11.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2443 12 1832 1820 0.0465554 0 - 0.957344 2024-04-16T15:41:12.815 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2444 12 1832 1820 0.0465363 0 - 0.957344 2024-04-16T15:41:13.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2445 12 1832 1820 0.0465173 0 - 0.957344 2024-04-16T15:41:14.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2446 12 1832 1820 0.0464983 0 - 0.957344 2024-04-16T15:41:15.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2447 12 1832 1820 0.0464793 0 - 0.957344 2024-04-16T15:41:16.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2448 12 1832 1820 0.0464603 0 - 0.957344 2024-04-16T15:41:17.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2449 12 1832 1820 0.0464413 0 - 0.957344 2024-04-16T15:41:18.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2450 12 1832 1820 0.0464224 0 - 0.957344 2024-04-16T15:41:19.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2451 12 1832 1820 0.0464034 0 - 0.957344 2024-04-16T15:41:20.816 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2452 12 1832 1820 0.0463845 0 - 0.957344 2024-04-16T15:41:21.817 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2453 12 1832 1820 0.0463656 0 - 0.957344 2024-04-16T15:41:22.817 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2454 12 1832 1820 0.0463467 0 - 0.957344 2024-04-16T15:41:23.817 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2455 12 1832 1820 0.0463278 0 - 0.957344 2024-04-16T15:41:24.817 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2456 12 1832 1820 0.046309 0 - 0.957344 2024-04-16T15:41:25.817 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2457 12 1832 1820 0.0462901 0 - 0.957344 2024-04-16T15:41:26.817 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2458 12 1832 1820 0.0462713 0 - 0.957344 2024-04-16T15:41:27.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2459 12 1832 1820 0.0462525 0 - 0.957344 2024-04-16T15:41:28.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:41:28.820442+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:41:28.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:41:28.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2460 12 1832 1820 0.0462337 0 - 0.957344 2024-04-16T15:41:29.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2461 12 1832 1820 0.0462149 0 - 0.957344 2024-04-16T15:41:30.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2462 12 1832 1820 0.0461961 0 - 0.957344 2024-04-16T15:41:31.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2463 12 1832 1820 0.0461774 0 - 0.957344 2024-04-16T15:41:32.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2464 12 1832 1820 0.0461586 0 - 0.957344 2024-04-16T15:41:33.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2465 12 1832 1820 0.0461399 0 - 0.957344 2024-04-16T15:41:34.818 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2466 12 1832 1820 0.0461212 0 - 0.957344 2024-04-16T15:41:35.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2467 12 1832 1820 0.0461025 0 - 0.957344 2024-04-16T15:41:36.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2468 12 1832 1820 0.0460838 0 - 0.957344 2024-04-16T15:41:37.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2469 12 1832 1820 0.0460651 0 - 0.957344 2024-04-16T15:41:38.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2470 12 1832 1820 0.0460465 0 - 0.957344 2024-04-16T15:41:39.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2471 12 1832 1820 0.0460278 0 - 0.957344 2024-04-16T15:41:40.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2472 12 1832 1820 0.0460092 0 - 0.957344 2024-04-16T15:41:41.819 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2473 12 1832 1820 0.0459906 0 - 0.957344 2024-04-16T15:41:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:41:42.595 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:41:42.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2474 12 1832 1820 0.045972 0 - 0.957344 2024-04-16T15:41:43.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2475 12 1832 1820 0.0459535 0 - 0.957344 2024-04-16T15:41:44.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:41:44.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:41:44.718+0000 7f4bcd8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:41:44.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2476 12 1832 1820 0.0459349 0 - 0.957344 2024-04-16T15:41:45.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2477 12 1832 1820 0.0459164 0 - 0.957344 2024-04-16T15:41:46.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2478 12 1832 1820 0.0458978 0 - 0.957344 2024-04-16T15:41:47.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2479 12 1832 1820 0.0458793 0 - 0.957344 2024-04-16T15:41:48.820 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:41:48.823219+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:41:48.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:41:48.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2480 12 1832 1820 0.0458608 0 - 0.957344 2024-04-16T15:41:49.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2481 12 1832 1820 0.0458423 0 - 0.957344 2024-04-16T15:41:50.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2482 12 1832 1820 0.0458239 0 - 0.957344 2024-04-16T15:41:51.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2483 12 1832 1820 0.0458054 0 - 0.957344 2024-04-16T15:41:52.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2484 12 1832 1820 0.045787 0 - 0.957344 2024-04-16T15:41:53.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2485 12 1832 1820 0.0457685 0 - 0.957344 2024-04-16T15:41:54.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2486 12 1832 1820 0.0457501 0 - 0.957344 2024-04-16T15:41:55.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2487 12 1832 1820 0.0457317 0 - 0.957344 2024-04-16T15:41:56.821 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2488 12 1832 1820 0.0457133 0 - 0.957344 2024-04-16T15:41:57.822 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2489 12 1832 1820 0.045695 0 - 0.957344 2024-04-16T15:41:58.822 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2490 12 1832 1820 0.0456766 0 - 0.957344 2024-04-16T15:41:59.822 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2491 12 1832 1820 0.0456583 0 - 0.957344 2024-04-16T15:42:00.822 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2492 12 1832 1820 0.04564 0 - 0.957344 2024-04-16T15:42:01.822 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2493 12 1832 1820 0.0456217 0 - 0.957344 2024-04-16T15:42:02.822 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2494 12 1832 1820 0.0456034 0 - 0.957344 2024-04-16T15:42:03.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2495 12 1832 1820 0.0455851 0 - 0.957344 2024-04-16T15:42:04.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2496 12 1832 1820 0.0455668 0 - 0.957344 2024-04-16T15:42:05.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2497 12 1832 1820 0.0455486 0 - 0.957344 2024-04-16T15:42:06.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2498 12 1832 1820 0.0455303 0 - 0.957344 2024-04-16T15:42:07.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2499 12 1832 1820 0.0455121 0 - 0.957344 2024-04-16T15:42:08.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:42:08.825914+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:42:08.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:42:08.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2500 12 1832 1820 0.0454939 0 - 0.957344 2024-04-16T15:42:09.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2501 12 1832 1820 0.0454757 0 - 0.957344 2024-04-16T15:42:10.823 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2502 12 1832 1820 0.0454576 0 - 0.957344 2024-04-16T15:42:11.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2503 12 1832 1820 0.0454394 0 - 0.957344 2024-04-16T15:42:12.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2504 12 1832 1820 0.0454213 0 - 0.957344 2024-04-16T15:42:13.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2505 12 1832 1820 0.0454031 0 - 0.957344 2024-04-16T15:42:14.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2506 12 1832 1820 0.045385 0 - 0.957344 2024-04-16T15:42:15.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2507 12 1832 1820 0.0453669 0 - 0.957344 2024-04-16T15:42:16.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2508 12 1832 1820 0.0453488 0 - 0.957344 2024-04-16T15:42:17.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2509 12 1832 1820 0.0453307 0 - 0.957344 2024-04-16T15:42:18.824 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2510 12 1832 1820 0.0453127 0 - 0.957344 2024-04-16T15:42:19.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2511 12 1832 1820 0.0452946 0 - 0.957344 2024-04-16T15:42:20.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2512 12 1832 1820 0.0452766 0 - 0.957344 2024-04-16T15:42:21.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2513 12 1832 1820 0.0452586 0 - 0.957344 2024-04-16T15:42:22.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2514 12 1832 1820 0.0452406 0 - 0.957344 2024-04-16T15:42:23.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2515 12 1832 1820 0.0452226 0 - 0.957344 2024-04-16T15:42:24.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2516 12 1832 1820 0.0452046 0 - 0.957344 2024-04-16T15:42:25.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2517 12 1832 1820 0.0451867 0 - 0.957344 2024-04-16T15:42:26.825 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2518 12 1832 1820 0.0451687 0 - 0.957344 2024-04-16T15:42:27.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2519 12 1832 1820 0.0451508 0 - 0.957344 2024-04-16T15:42:28.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:42:28.828567+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:42:28.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:42:28.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2520 12 1832 1820 0.0451329 0 - 0.957344 2024-04-16T15:42:29.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2521 12 1832 1820 0.045115 0 - 0.957344 2024-04-16T15:42:30.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2522 12 1832 1820 0.0450971 0 - 0.957344 2024-04-16T15:42:31.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2523 12 1832 1820 0.0450792 0 - 0.957344 2024-04-16T15:42:32.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2524 12 1832 1820 0.0450613 0 - 0.957344 2024-04-16T15:42:33.826 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2525 12 1832 1820 0.0450435 0 - 0.957344 2024-04-16T15:42:34.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2526 12 1832 1820 0.0450257 0 - 0.957344 2024-04-16T15:42:35.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2527 12 1832 1820 0.0450078 0 - 0.957344 2024-04-16T15:42:36.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2528 12 1832 1820 0.04499 0 - 0.957344 2024-04-16T15:42:37.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2529 12 1832 1820 0.0449722 0 - 0.957344 2024-04-16T15:42:38.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2530 12 1832 1820 0.0449545 0 - 0.957344 2024-04-16T15:42:39.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2531 12 1832 1820 0.0449367 0 - 0.957344 2024-04-16T15:42:40.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2532 12 1832 1820 0.044919 0 - 0.957344 2024-04-16T15:42:41.827 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2533 12 1832 1820 0.0449012 0 - 0.957344 2024-04-16T15:42:42.828 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2534 12 1832 1820 0.0448835 0 - 0.957344 2024-04-16T15:42:43.828 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2535 12 1832 1820 0.0448658 0 - 0.957344 2024-04-16T15:42:44.828 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2536 12 1832 1820 0.0448481 0 - 0.957344 2024-04-16T15:42:45.828 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2537 12 1832 1820 0.0448304 0 - 0.957344 2024-04-16T15:42:46.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2538 12 1832 1820 0.0448128 0 - 0.957344 2024-04-16T15:42:47.828 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2539 12 1832 1820 0.0447951 0 - 0.957344 2024-04-16T15:42:48.828 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:42:48.831162+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:42:48.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:42:48.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2540 12 1832 1820 0.0447775 0 - 0.957344 2024-04-16T15:42:49.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2541 12 1832 1820 0.0447599 0 - 0.957344 2024-04-16T15:42:50.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2542 12 1832 1820 0.0447423 0 - 0.957344 2024-04-16T15:42:51.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2543 12 1832 1820 0.0447247 0 - 0.957344 2024-04-16T15:42:52.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2544 12 1832 1820 0.0447071 0 - 0.957344 2024-04-16T15:42:53.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2545 12 1832 1820 0.0446895 0 - 0.957344 2024-04-16T15:42:54.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2546 12 1832 1820 0.044672 0 - 0.957344 2024-04-16T15:42:55.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2547 12 1832 1820 0.0446544 0 - 0.957344 2024-04-16T15:42:56.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2548 12 1832 1820 0.0446369 0 - 0.957344 2024-04-16T15:42:57.829 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2549 12 1832 1820 0.0446194 0 - 0.957344 2024-04-16T15:42:58.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2550 12 1832 1820 0.0446019 0 - 0.957344 2024-04-16T15:42:59.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2551 12 1832 1820 0.0445844 0 - 0.957344 2024-04-16T15:43:00.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2552 12 1832 1820 0.0445669 0 - 0.957344 2024-04-16T15:43:01.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2553 12 1832 1820 0.0445495 0 - 0.957344 2024-04-16T15:43:02.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2554 12 1832 1820 0.044532 0 - 0.957344 2024-04-16T15:43:03.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2555 12 1832 1820 0.0445146 0 - 0.957344 2024-04-16T15:43:04.830 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2556 12 1832 1820 0.0444972 0 - 0.957344 2024-04-16T15:43:05.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2557 12 1832 1820 0.0444798 0 - 0.957344 2024-04-16T15:43:06.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2558 12 1832 1820 0.0444624 0 - 0.957344 2024-04-16T15:43:07.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2559 12 1832 1820 0.044445 0 - 0.957344 2024-04-16T15:43:08.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:43:08.833876+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:43:08.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:43:08.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2560 12 1832 1820 0.0444277 0 - 0.957344 2024-04-16T15:43:09.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2561 12 1832 1820 0.0444103 0 - 0.957344 2024-04-16T15:43:10.831 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2562 12 1832 1820 0.044393 0 - 0.957344 2024-04-16T15:43:11.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2563 12 1832 1820 0.0443757 0 - 0.957344 2024-04-16T15:43:12.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2564 12 1832 1820 0.0443584 0 - 0.957344 2024-04-16T15:43:13.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2565 12 1832 1820 0.0443411 0 - 0.957344 2024-04-16T15:43:14.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2566 12 1832 1820 0.0443238 0 - 0.957344 2024-04-16T15:43:15.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2567 12 1832 1820 0.0443065 0 - 0.957344 2024-04-16T15:43:16.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2568 12 1832 1820 0.0442893 0 - 0.957344 2024-04-16T15:43:17.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2569 12 1832 1820 0.044272 0 - 0.957344 2024-04-16T15:43:18.832 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2570 12 1832 1820 0.0442548 0 - 0.957344 2024-04-16T15:43:19.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2571 12 1832 1820 0.0442376 0 - 0.957344 2024-04-16T15:43:20.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2572 12 1832 1820 0.0442204 0 - 0.957344 2024-04-16T15:43:21.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2573 12 1832 1820 0.0442032 0 - 0.957344 2024-04-16T15:43:22.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2574 12 1832 1820 0.044186 0 - 0.957344 2024-04-16T15:43:23.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2575 12 1832 1820 0.0441689 0 - 0.957344 2024-04-16T15:43:24.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2576 12 1832 1820 0.0441517 0 - 0.957344 2024-04-16T15:43:25.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2577 12 1832 1820 0.0441346 0 - 0.957344 2024-04-16T15:43:26.833 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2578 12 1832 1820 0.0441175 0 - 0.957344 2024-04-16T15:43:27.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2579 12 1832 1820 0.0441004 0 - 0.957344 2024-04-16T15:43:28.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:43:28.836551+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:43:28.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:43:28.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2580 12 1832 1820 0.0440833 0 - 0.957344 2024-04-16T15:43:29.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2581 12 1832 1820 0.0440662 0 - 0.957344 2024-04-16T15:43:30.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2582 12 1832 1820 0.0440491 0 - 0.957344 2024-04-16T15:43:31.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2583 12 1832 1820 0.0440321 0 - 0.957344 2024-04-16T15:43:32.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2584 12 1832 1820 0.044015 0 - 0.957344 2024-04-16T15:43:33.834 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2585 12 1832 1820 0.043998 0 - 0.957344 2024-04-16T15:43:34.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2586 12 1832 1820 0.043981 0 - 0.957344 2024-04-16T15:43:35.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2587 12 1832 1820 0.043964 0 - 0.957344 2024-04-16T15:43:36.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2588 12 1832 1820 0.043947 0 - 0.957344 2024-04-16T15:43:37.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2589 12 1832 1820 0.04393 0 - 0.957344 2024-04-16T15:43:38.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2590 12 1832 1820 0.0439131 0 - 0.957344 2024-04-16T15:43:39.835 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2591 12 1832 1820 0.0438961 0 - 0.957344 2024-04-16T15:43:40.836 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2592 12 1832 1820 0.0438792 0 - 0.957344 2024-04-16T15:43:41.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2593 12 1832 1820 0.0438623 0 - 0.957344 2024-04-16T15:43:42.836 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2594 12 1832 1820 0.0438453 0 - 0.957344 2024-04-16T15:43:43.836 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2595 12 1832 1820 0.0438284 0 - 0.957344 2024-04-16T15:43:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:43:44.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:43:44.836 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2596 12 1832 1820 0.0438116 0 - 0.957344 2024-04-16T15:43:45.836 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2597 12 1832 1820 0.0437947 0 - 0.957344 2024-04-16T15:43:46.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:43:46.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:43:46.760+0000 7fc78b55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:43:46.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2598 12 1832 1820 0.0437778 0 - 0.957344 2024-04-16T15:43:47.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2599 12 1832 1820 0.043761 0 - 0.957344 2024-04-16T15:43:48.836 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:43:48.839187+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:43:48.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:43:48.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2600 12 1832 1820 0.0437442 0 - 0.957344 2024-04-16T15:43:49.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2601 12 1832 1820 0.0437273 0 - 0.957344 2024-04-16T15:43:50.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2602 12 1832 1820 0.0437105 0 - 0.957344 2024-04-16T15:43:51.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2603 12 1832 1820 0.0436937 0 - 0.957344 2024-04-16T15:43:52.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2604 12 1832 1820 0.043677 0 - 0.957344 2024-04-16T15:43:53.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2605 12 1832 1820 0.0436602 0 - 0.957344 2024-04-16T15:43:54.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2606 12 1832 1820 0.0436434 0 - 0.957344 2024-04-16T15:43:55.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2607 12 1832 1820 0.0436267 0 - 0.957344 2024-04-16T15:43:56.837 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2608 12 1832 1820 0.04361 0 - 0.957344 2024-04-16T15:43:57.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2609 12 1832 1820 0.0435933 0 - 0.957344 2024-04-16T15:43:58.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2610 12 1832 1820 0.0435766 0 - 0.957344 2024-04-16T15:43:59.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2611 12 1832 1820 0.0435599 0 - 0.957344 2024-04-16T15:44:00.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2612 12 1832 1820 0.0435432 0 - 0.957344 2024-04-16T15:44:01.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2613 12 1832 1820 0.0435265 0 - 0.957344 2024-04-16T15:44:02.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2614 12 1832 1820 0.0435099 0 - 0.957344 2024-04-16T15:44:03.838 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2615 12 1832 1820 0.0434932 0 - 0.957344 2024-04-16T15:44:04.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2616 12 1832 1820 0.0434766 0 - 0.957344 2024-04-16T15:44:05.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2617 12 1832 1820 0.04346 0 - 0.957344 2024-04-16T15:44:06.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2618 12 1832 1820 0.0434434 0 - 0.957344 2024-04-16T15:44:07.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2619 12 1832 1820 0.0434268 0 - 0.957344 2024-04-16T15:44:08.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:44:08.841878+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:44:08.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:44:08.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2620 12 1832 1820 0.0434102 0 - 0.957344 2024-04-16T15:44:09.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2621 12 1832 1820 0.0433937 0 - 0.957344 2024-04-16T15:44:10.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2622 12 1832 1820 0.0433771 0 - 0.957344 2024-04-16T15:44:11.839 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2623 12 1832 1820 0.0433606 0 - 0.957344 2024-04-16T15:44:12.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2624 12 1832 1820 0.0433441 0 - 0.957344 2024-04-16T15:44:13.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2625 12 1832 1820 0.0433275 0 - 0.957344 2024-04-16T15:44:14.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2626 12 1832 1820 0.043311 0 - 0.957344 2024-04-16T15:44:15.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2627 12 1832 1820 0.0432946 0 - 0.957344 2024-04-16T15:44:16.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2628 12 1832 1820 0.0432781 0 - 0.957344 2024-04-16T15:44:17.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2629 12 1832 1820 0.0432616 0 - 0.957344 2024-04-16T15:44:18.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2630 12 1832 1820 0.0432452 0 - 0.957344 2024-04-16T15:44:19.840 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2631 12 1832 1820 0.0432287 0 - 0.957344 2024-04-16T15:44:20.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2632 12 1832 1820 0.0432123 0 - 0.957344 2024-04-16T15:44:21.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2633 12 1832 1820 0.0431959 0 - 0.957344 2024-04-16T15:44:22.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2634 12 1832 1820 0.0431795 0 - 0.957344 2024-04-16T15:44:23.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2635 12 1832 1820 0.0431631 0 - 0.957344 2024-04-16T15:44:24.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2636 12 1832 1820 0.0431467 0 - 0.957344 2024-04-16T15:44:25.841 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2637 12 1832 1820 0.0431304 0 - 0.957344 2024-04-16T15:44:26.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2638 12 1832 1820 0.043114 0 - 0.957344 2024-04-16T15:44:27.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2639 12 1832 1820 0.0430977 0 - 0.957344 2024-04-16T15:44:28.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:44:28.844604+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:44:28.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:44:28.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2640 12 1832 1820 0.0430814 0 - 0.957344 2024-04-16T15:44:29.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2641 12 1832 1820 0.0430651 0 - 0.957344 2024-04-16T15:44:30.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2642 12 1832 1820 0.0430488 0 - 0.957344 2024-04-16T15:44:31.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2643 12 1832 1820 0.0430325 0 - 0.957344 2024-04-16T15:44:32.842 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2644 12 1832 1820 0.0430162 0 - 0.957344 2024-04-16T15:44:33.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2645 12 1832 1820 0.0429999 0 - 0.957344 2024-04-16T15:44:34.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2646 12 1832 1820 0.0429837 0 - 0.957344 2024-04-16T15:44:35.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2647 12 1832 1820 0.0429674 0 - 0.957344 2024-04-16T15:44:36.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2648 12 1832 1820 0.0429512 0 - 0.957344 2024-04-16T15:44:37.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2649 12 1832 1820 0.042935 0 - 0.957344 2024-04-16T15:44:38.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2650 12 1832 1820 0.0429188 0 - 0.957344 2024-04-16T15:44:39.843 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2651 12 1832 1820 0.0429026 0 - 0.957344 2024-04-16T15:44:40.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2652 12 1832 1820 0.0428864 0 - 0.957344 2024-04-16T15:44:41.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2653 12 1832 1820 0.0428703 0 - 0.957344 2024-04-16T15:44:42.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2654 12 1832 1820 0.0428541 0 - 0.957344 2024-04-16T15:44:43.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2655 12 1832 1820 0.042838 0 - 0.957344 2024-04-16T15:44:44.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2656 12 1832 1820 0.0428218 0 - 0.957344 2024-04-16T15:44:45.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2657 12 1832 1820 0.0428057 0 - 0.957344 2024-04-16T15:44:46.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2658 12 1832 1820 0.0427896 0 - 0.957344 2024-04-16T15:44:47.844 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2659 12 1832 1820 0.0427735 0 - 0.957344 2024-04-16T15:44:48.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:44:48.847280+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:44:48.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:44:48.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2660 12 1832 1820 0.0427574 0 - 0.957344 2024-04-16T15:44:49.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2661 12 1832 1820 0.0427414 0 - 0.957344 2024-04-16T15:44:50.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2662 12 1832 1820 0.0427253 0 - 0.957344 2024-04-16T15:44:51.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2663 12 1832 1820 0.0427093 0 - 0.957344 2024-04-16T15:44:52.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2664 12 1832 1820 0.0426932 0 - 0.957344 2024-04-16T15:44:53.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2665 12 1832 1820 0.0426772 0 - 0.957344 2024-04-16T15:44:54.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2666 12 1832 1820 0.0426612 0 - 0.957344 2024-04-16T15:44:55.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2667 12 1832 1820 0.0426452 0 - 0.957344 2024-04-16T15:44:56.845 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2668 12 1832 1820 0.0426292 0 - 0.957344 2024-04-16T15:44:57.846 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2669 12 1832 1820 0.0426133 0 - 0.957344 2024-04-16T15:44:58.846 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2670 12 1832 1820 0.0425973 0 - 0.957344 2024-04-16T15:44:59.846 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2671 12 1832 1820 0.0425814 0 - 0.957344 2024-04-16T15:45:00.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2672 12 1832 1820 0.0425654 0 - 0.957344 2024-04-16T15:45:01.846 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2673 12 1832 1820 0.0425495 0 - 0.957344 2024-04-16T15:45:02.846 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2674 12 1832 1820 0.0425336 0 - 0.957344 2024-04-16T15:45:03.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2675 12 1832 1820 0.0425177 0 - 0.957344 2024-04-16T15:45:04.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2676 12 1832 1820 0.0425018 0 - 0.957344 2024-04-16T15:45:05.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2677 12 1832 1820 0.0424859 0 - 0.957344 2024-04-16T15:45:06.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2678 12 1832 1820 0.0424701 0 - 0.957344 2024-04-16T15:45:07.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2679 12 1832 1820 0.0424542 0 - 0.957344 2024-04-16T15:45:08.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:45:08.849996+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:45:08.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:45:08.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2680 12 1832 1820 0.0424384 0 - 0.957344 2024-04-16T15:45:09.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2681 12 1832 1820 0.0424225 0 - 0.957344 2024-04-16T15:45:10.847 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2682 12 1832 1820 0.0424067 0 - 0.957344 2024-04-16T15:45:11.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2683 12 1832 1820 0.0423909 0 - 0.957344 2024-04-16T15:45:12.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2684 12 1832 1820 0.0423751 0 - 0.957344 2024-04-16T15:45:13.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2685 12 1832 1820 0.0423593 0 - 0.957344 2024-04-16T15:45:14.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2686 12 1832 1820 0.0423436 0 - 0.957344 2024-04-16T15:45:15.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2687 12 1832 1820 0.0423278 0 - 0.957344 2024-04-16T15:45:16.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2688 12 1832 1820 0.0423121 0 - 0.957344 2024-04-16T15:45:17.848 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2689 12 1832 1820 0.0422963 0 - 0.957344 2024-04-16T15:45:18.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2690 12 1832 1820 0.0422806 0 - 0.957344 2024-04-16T15:45:19.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2691 12 1832 1820 0.0422649 0 - 0.957344 2024-04-16T15:45:20.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2692 12 1832 1820 0.0422492 0 - 0.957344 2024-04-16T15:45:21.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2693 12 1832 1820 0.0422335 0 - 0.957344 2024-04-16T15:45:22.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2694 12 1832 1820 0.0422178 0 - 0.957344 2024-04-16T15:45:23.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2695 12 1832 1820 0.0422022 0 - 0.957344 2024-04-16T15:45:24.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2696 12 1832 1820 0.0421865 0 - 0.957344 2024-04-16T15:45:25.849 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2697 12 1832 1820 0.0421709 0 - 0.957344 2024-04-16T15:45:26.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2698 12 1832 1820 0.0421552 0 - 0.957344 2024-04-16T15:45:27.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2699 12 1832 1820 0.0421396 0 - 0.957344 2024-04-16T15:45:28.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:45:28.852622+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:45:28.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:45:28.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2700 12 1832 1820 0.042124 0 - 0.957344 2024-04-16T15:45:29.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2701 12 1832 1820 0.0421084 0 - 0.957344 2024-04-16T15:45:30.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2702 12 1832 1820 0.0420928 0 - 0.957344 2024-04-16T15:45:31.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2703 12 1832 1820 0.0420773 0 - 0.957344 2024-04-16T15:45:32.850 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2704 12 1832 1820 0.0420617 0 - 0.957344 2024-04-16T15:45:33.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2705 12 1832 1820 0.0420461 0 - 0.957344 2024-04-16T15:45:34.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2706 12 1832 1820 0.0420306 0 - 0.957344 2024-04-16T15:45:35.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2707 12 1832 1820 0.0420151 0 - 0.957344 2024-04-16T15:45:36.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2708 12 1832 1820 0.0419996 0 - 0.957344 2024-04-16T15:45:37.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2709 12 1832 1820 0.0419841 0 - 0.957344 2024-04-16T15:45:38.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2710 12 1832 1820 0.0419686 0 - 0.957344 2024-04-16T15:45:39.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2711 12 1832 1820 0.0419531 0 - 0.957344 2024-04-16T15:45:40.851 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2712 12 1832 1820 0.0419376 0 - 0.957344 2024-04-16T15:45:41.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2713 12 1832 1820 0.0419222 0 - 0.957344 2024-04-16T15:45:42.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2714 12 1832 1820 0.0419067 0 - 0.957344 2024-04-16T15:45:43.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2715 12 1832 1820 0.0418913 0 - 0.957344 2024-04-16T15:45:44.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2716 12 1832 1820 0.0418759 0 - 0.957344 2024-04-16T15:45:45.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2717 12 1832 1820 0.0418604 0 - 0.957344 2024-04-16T15:45:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:45:46.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:45:46.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2718 12 1832 1820 0.041845 0 - 0.957344 2024-04-16T15:45:47.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2719 12 1832 1820 0.0418296 0 - 0.957344 2024-04-16T15:45:48.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:45:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:45:48.800+0000 7f0ad266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:45:48.852 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:45:48.855223+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:45:48.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:45:48.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2720 12 1832 1820 0.0418143 0 - 0.957344 2024-04-16T15:45:49.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2721 12 1832 1820 0.0417989 0 - 0.957344 2024-04-16T15:45:50.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2722 12 1832 1820 0.0417835 0 - 0.957344 2024-04-16T15:45:51.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2723 12 1832 1820 0.0417682 0 - 0.957344 2024-04-16T15:45:52.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2724 12 1832 1820 0.0417529 0 - 0.957344 2024-04-16T15:45:53.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2725 12 1832 1820 0.0417375 0 - 0.957344 2024-04-16T15:45:54.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2726 12 1832 1820 0.0417222 0 - 0.957344 2024-04-16T15:45:55.853 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2727 12 1832 1820 0.0417069 0 - 0.957344 2024-04-16T15:45:56.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2728 12 1832 1820 0.0416916 0 - 0.957344 2024-04-16T15:45:57.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2729 12 1832 1820 0.0416764 0 - 0.957344 2024-04-16T15:45:58.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2730 12 1832 1820 0.0416611 0 - 0.957344 2024-04-16T15:45:59.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2731 12 1832 1820 0.0416459 0 - 0.957344 2024-04-16T15:46:00.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2732 12 1832 1820 0.0416306 0 - 0.957344 2024-04-16T15:46:01.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2733 12 1832 1820 0.0416154 0 - 0.957344 2024-04-16T15:46:02.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2734 12 1832 1820 0.0416002 0 - 0.957344 2024-04-16T15:46:03.854 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2735 12 1832 1820 0.0415849 0 - 0.957344 2024-04-16T15:46:04.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2736 12 1832 1820 0.0415697 0 - 0.957344 2024-04-16T15:46:05.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2737 12 1832 1820 0.0415546 0 - 0.957344 2024-04-16T15:46:06.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2738 12 1832 1820 0.0415394 0 - 0.957344 2024-04-16T15:46:07.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2739 12 1832 1820 0.0415242 0 - 0.957344 2024-04-16T15:46:08.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:46:08.857807+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:46:08.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:46:08.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2740 12 1832 1820 0.0415091 0 - 0.957344 2024-04-16T15:46:09.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2741 12 1832 1820 0.0414939 0 - 0.957344 2024-04-16T15:46:10.855 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2742 12 1832 1820 0.0414788 0 - 0.957344 2024-04-16T15:46:11.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2743 12 1832 1820 0.0414637 0 - 0.957344 2024-04-16T15:46:12.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2744 12 1832 1820 0.0414485 0 - 0.957344 2024-04-16T15:46:13.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2745 12 1832 1820 0.0414335 0 - 0.957344 2024-04-16T15:46:14.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2746 12 1832 1820 0.0414184 0 - 0.957344 2024-04-16T15:46:15.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2747 12 1832 1820 0.0414033 0 - 0.957344 2024-04-16T15:46:16.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2748 12 1832 1820 0.0413882 0 - 0.957344 2024-04-16T15:46:17.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2749 12 1832 1820 0.0413732 0 - 0.957344 2024-04-16T15:46:18.856 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2750 12 1832 1820 0.0413581 0 - 0.957344 2024-04-16T15:46:19.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2751 12 1832 1820 0.0413431 0 - 0.957344 2024-04-16T15:46:20.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2752 12 1832 1820 0.0413281 0 - 0.957344 2024-04-16T15:46:21.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2753 12 1832 1820 0.041313 0 - 0.957344 2024-04-16T15:46:22.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2754 12 1832 1820 0.041298 0 - 0.957344 2024-04-16T15:46:23.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2755 12 1832 1820 0.0412831 0 - 0.957344 2024-04-16T15:46:24.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2756 12 1832 1820 0.0412681 0 - 0.957344 2024-04-16T15:46:25.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2757 12 1832 1820 0.0412531 0 - 0.957344 2024-04-16T15:46:26.857 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2758 12 1832 1820 0.0412382 0 - 0.957344 2024-04-16T15:46:27.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2759 12 1832 1820 0.0412232 0 - 0.957344 2024-04-16T15:46:28.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:46:28.860492+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:46:28.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:46:28.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2760 12 1832 1820 0.0412083 0 - 0.957344 2024-04-16T15:46:29.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2761 12 1832 1820 0.0411933 0 - 0.957344 2024-04-16T15:46:30.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2762 12 1832 1820 0.0411784 0 - 0.957344 2024-04-16T15:46:31.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2763 12 1832 1820 0.0411635 0 - 0.957344 2024-04-16T15:46:32.858 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2764 12 1832 1820 0.0411486 0 - 0.957344 2024-04-16T15:46:33.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2765 12 1832 1820 0.0411337 0 - 0.957344 2024-04-16T15:46:34.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2766 12 1832 1820 0.0411189 0 - 0.957344 2024-04-16T15:46:35.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2767 12 1832 1820 0.041104 0 - 0.957344 2024-04-16T15:46:36.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2768 12 1832 1820 0.0410892 0 - 0.957344 2024-04-16T15:46:37.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2769 12 1832 1820 0.0410743 0 - 0.957344 2024-04-16T15:46:38.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2770 12 1832 1820 0.0410595 0 - 0.957344 2024-04-16T15:46:39.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2771 12 1832 1820 0.0410447 0 - 0.957344 2024-04-16T15:46:40.859 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2772 12 1832 1820 0.0410299 0 - 0.957344 2024-04-16T15:46:41.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2773 12 1832 1820 0.0410151 0 - 0.957344 2024-04-16T15:46:42.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2774 12 1832 1820 0.0410003 0 - 0.957344 2024-04-16T15:46:43.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2775 12 1832 1820 0.0409855 0 - 0.957344 2024-04-16T15:46:44.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2776 12 1832 1820 0.0409708 0 - 0.957344 2024-04-16T15:46:45.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2777 12 1832 1820 0.040956 0 - 0.957344 2024-04-16T15:46:46.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2778 12 1832 1820 0.0409413 0 - 0.957344 2024-04-16T15:46:47.860 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2779 12 1832 1820 0.0409265 0 - 0.957344 2024-04-16T15:46:48.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:46:48.863209+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:46:48.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:46:48.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2780 12 1832 1820 0.0409118 0 - 0.957344 2024-04-16T15:46:49.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2781 12 1832 1820 0.0408971 0 - 0.957344 2024-04-16T15:46:50.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2782 12 1832 1820 0.0408824 0 - 0.957344 2024-04-16T15:46:51.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2783 12 1832 1820 0.0408677 0 - 0.957344 2024-04-16T15:46:52.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2784 12 1832 1820 0.040853 0 - 0.957344 2024-04-16T15:46:53.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2785 12 1832 1820 0.0408384 0 - 0.957344 2024-04-16T15:46:54.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2786 12 1832 1820 0.0408237 0 - 0.957344 2024-04-16T15:46:55.861 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2787 12 1832 1820 0.040809 0 - 0.957344 2024-04-16T15:46:56.862 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2788 12 1832 1820 0.0407944 0 - 0.957344 2024-04-16T15:46:57.862 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2789 12 1832 1820 0.0407798 0 - 0.957344 2024-04-16T15:46:58.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2790 12 1832 1820 0.0407652 0 - 0.957344 2024-04-16T15:46:59.862 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2791 12 1832 1820 0.0407506 0 - 0.957344 2024-04-16T15:47:00.862 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2792 12 1832 1820 0.040736 0 - 0.957344 2024-04-16T15:47:01.862 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2793 12 1832 1820 0.0407214 0 - 0.957344 2024-04-16T15:47:02.862 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2794 12 1832 1820 0.0407068 0 - 0.957344 2024-04-16T15:47:03.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2795 12 1832 1820 0.0406922 0 - 0.957344 2024-04-16T15:47:04.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2796 12 1832 1820 0.0406777 0 - 0.957344 2024-04-16T15:47:05.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2797 12 1832 1820 0.0406631 0 - 0.957344 2024-04-16T15:47:06.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2798 12 1832 1820 0.0406486 0 - 0.957344 2024-04-16T15:47:07.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2799 12 1832 1820 0.0406341 0 - 0.957344 2024-04-16T15:47:08.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:47:08.866010+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:47:08.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:47:08.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2800 12 1832 1820 0.0406196 0 - 0.957344 2024-04-16T15:47:09.863 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2801 12 1832 1820 0.0406051 0 - 0.957344 2024-04-16T15:47:10.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2802 12 1832 1820 0.0405906 0 - 0.957344 2024-04-16T15:47:11.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2803 12 1832 1820 0.0405761 0 - 0.957344 2024-04-16T15:47:12.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2804 12 1832 1820 0.0405616 0 - 0.957344 2024-04-16T15:47:13.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2805 12 1832 1820 0.0405472 0 - 0.957344 2024-04-16T15:47:14.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2806 12 1832 1820 0.0405327 0 - 0.957344 2024-04-16T15:47:15.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2807 12 1832 1820 0.0405183 0 - 0.957344 2024-04-16T15:47:16.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2808 12 1832 1820 0.0405039 0 - 0.957344 2024-04-16T15:47:17.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2809 12 1832 1820 0.0404894 0 - 0.957344 2024-04-16T15:47:18.864 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2810 12 1832 1820 0.040475 0 - 0.957344 2024-04-16T15:47:19.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2811 12 1832 1820 0.0404606 0 - 0.957344 2024-04-16T15:47:20.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2812 12 1832 1820 0.0404462 0 - 0.957344 2024-04-16T15:47:21.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2813 12 1832 1820 0.0404319 0 - 0.957344 2024-04-16T15:47:22.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2814 12 1832 1820 0.0404175 0 - 0.957344 2024-04-16T15:47:23.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2815 12 1832 1820 0.0404031 0 - 0.957344 2024-04-16T15:47:24.865 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2816 12 1832 1820 0.0403888 0 - 0.957344 2024-04-16T15:47:25.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2817 12 1832 1820 0.0403744 0 - 0.957344 2024-04-16T15:47:26.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2818 12 1832 1820 0.0403601 0 - 0.957344 2024-04-16T15:47:27.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2819 12 1832 1820 0.0403458 0 - 0.957344 2024-04-16T15:47:28.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:47:28.868591+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:47:28.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:47:28.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2820 12 1832 1820 0.0403315 0 - 0.957344 2024-04-16T15:47:29.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2821 12 1832 1820 0.0403172 0 - 0.957344 2024-04-16T15:47:30.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2822 12 1832 1820 0.0403029 0 - 0.957344 2024-04-16T15:47:31.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2823 12 1832 1820 0.0402886 0 - 0.957344 2024-04-16T15:47:32.866 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2824 12 1832 1820 0.0402744 0 - 0.957344 2024-04-16T15:47:33.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2825 12 1832 1820 0.0402601 0 - 0.957344 2024-04-16T15:47:34.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2826 12 1832 1820 0.0402459 0 - 0.957344 2024-04-16T15:47:35.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2827 12 1832 1820 0.0402316 0 - 0.957344 2024-04-16T15:47:36.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2828 12 1832 1820 0.0402174 0 - 0.957344 2024-04-16T15:47:37.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2829 12 1832 1820 0.0402032 0 - 0.957344 2024-04-16T15:47:38.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2830 12 1832 1820 0.040189 0 - 0.957344 2024-04-16T15:47:39.867 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2831 12 1832 1820 0.0401748 0 - 0.957344 2024-04-16T15:47:40.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2832 12 1832 1820 0.0401606 0 - 0.957344 2024-04-16T15:47:41.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2833 12 1832 1820 0.0401464 0 - 0.957344 2024-04-16T15:47:42.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2834 12 1832 1820 0.0401323 0 - 0.957344 2024-04-16T15:47:43.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2835 12 1832 1820 0.0401181 0 - 0.957344 2024-04-16T15:47:44.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2836 12 1832 1820 0.040104 0 - 0.957344 2024-04-16T15:47:45.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2837 12 1832 1820 0.0400898 0 - 0.957344 2024-04-16T15:47:46.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2838 12 1832 1820 0.0400757 0 - 0.957344 2024-04-16T15:47:47.868 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2839 12 1832 1820 0.0400616 0 - 0.957344 2024-04-16T15:47:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:47:48.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:47:48.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:47:48.871228+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:47:48.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:47:48.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2840 12 1832 1820 0.0400475 0 - 0.957344 2024-04-16T15:47:49.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2841 12 1832 1820 0.0400334 0 - 0.957344 2024-04-16T15:47:50.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T15:47:50.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:47:50.841+0000 7fcf784ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:47:50.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2842 12 1832 1820 0.0400193 0 - 0.957344 2024-04-16T15:47:51.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2843 12 1832 1820 0.0400052 0 - 0.957344 2024-04-16T15:47:52.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2844 12 1832 1820 0.0399911 0 - 0.957344 2024-04-16T15:47:53.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2845 12 1832 1820 0.0399771 0 - 0.957344 2024-04-16T15:47:54.869 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2846 12 1832 1820 0.039963 0 - 0.957344 2024-04-16T15:47:55.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2847 12 1832 1820 0.039949 0 - 0.957344 2024-04-16T15:47:56.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2848 12 1832 1820 0.039935 0 - 0.957344 2024-04-16T15:47:57.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2849 12 1832 1820 0.039921 0 - 0.957344 2024-04-16T15:47:58.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2850 12 1832 1820 0.039907 0 - 0.957344 2024-04-16T15:47:59.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2851 12 1832 1820 0.039893 0 - 0.957344 2024-04-16T15:48:00.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2852 12 1832 1820 0.039879 0 - 0.957344 2024-04-16T15:48:01.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2853 12 1832 1820 0.039865 0 - 0.957344 2024-04-16T15:48:02.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2854 12 1832 1820 0.039851 0 - 0.957344 2024-04-16T15:48:03.870 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2855 12 1832 1820 0.0398371 0 - 0.957344 2024-04-16T15:48:04.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2856 12 1832 1820 0.0398231 0 - 0.957344 2024-04-16T15:48:05.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2857 12 1832 1820 0.0398092 0 - 0.957344 2024-04-16T15:48:06.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2858 12 1832 1820 0.0397953 0 - 0.957344 2024-04-16T15:48:07.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2859 12 1832 1820 0.0397813 0 - 0.957344 2024-04-16T15:48:08.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:48:08.873642+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:48:08.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:48:08.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2860 12 1832 1820 0.0397674 0 - 0.957344 2024-04-16T15:48:09.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2861 12 1832 1820 0.0397535 0 - 0.957344 2024-04-16T15:48:10.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2862 12 1832 1820 0.0397396 0 - 0.957344 2024-04-16T15:48:11.871 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2863 12 1832 1820 0.0397258 0 - 0.957344 2024-04-16T15:48:12.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2864 12 1832 1820 0.0397119 0 - 0.957344 2024-04-16T15:48:13.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2865 12 1832 1820 0.039698 0 - 0.957344 2024-04-16T15:48:14.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2866 12 1832 1820 0.0396842 0 - 0.957344 2024-04-16T15:48:15.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2867 12 1832 1820 0.0396703 0 - 0.957344 2024-04-16T15:48:16.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2868 12 1832 1820 0.0396565 0 - 0.957344 2024-04-16T15:48:17.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2869 12 1832 1820 0.0396427 0 - 0.957344 2024-04-16T15:48:18.872 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2870 12 1832 1820 0.0396289 0 - 0.957344 2024-04-16T15:48:19.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2871 12 1832 1820 0.0396151 0 - 0.957344 2024-04-16T15:48:20.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2872 12 1832 1820 0.0396013 0 - 0.957344 2024-04-16T15:48:21.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2873 12 1832 1820 0.0395875 0 - 0.957344 2024-04-16T15:48:22.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2874 12 1832 1820 0.0395737 0 - 0.957344 2024-04-16T15:48:23.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2875 12 1832 1820 0.0395599 0 - 0.957344 2024-04-16T15:48:24.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2876 12 1832 1820 0.0395462 0 - 0.957344 2024-04-16T15:48:25.873 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2877 12 1832 1820 0.0395324 0 - 0.957344 2024-04-16T15:48:26.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2878 12 1832 1820 0.0395187 0 - 0.957344 2024-04-16T15:48:27.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2879 12 1832 1820 0.039505 0 - 0.957344 2024-04-16T15:48:28.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:48:28.876422+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:48:28.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:48:28.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2880 12 1832 1820 0.0394913 0 - 0.957344 2024-04-16T15:48:29.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2881 12 1832 1820 0.0394776 0 - 0.957344 2024-04-16T15:48:30.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2882 12 1832 1820 0.0394639 0 - 0.957344 2024-04-16T15:48:31.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2883 12 1832 1820 0.0394502 0 - 0.957344 2024-04-16T15:48:32.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2884 12 1832 1820 0.0394365 0 - 0.957344 2024-04-16T15:48:33.874 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2885 12 1832 1820 0.0394228 0 - 0.957344 2024-04-16T15:48:34.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2886 12 1832 1820 0.0394092 0 - 0.957344 2024-04-16T15:48:35.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2887 12 1832 1820 0.0393955 0 - 0.957344 2024-04-16T15:48:36.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2888 12 1832 1820 0.0393819 0 - 0.957344 2024-04-16T15:48:37.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2889 12 1832 1820 0.0393682 0 - 0.957344 2024-04-16T15:48:38.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2890 12 1832 1820 0.0393546 0 - 0.957344 2024-04-16T15:48:39.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2891 12 1832 1820 0.039341 0 - 0.957344 2024-04-16T15:48:40.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2892 12 1832 1820 0.0393274 0 - 0.957344 2024-04-16T15:48:41.875 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2893 12 1832 1820 0.0393138 0 - 0.957344 2024-04-16T15:48:42.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2894 12 1832 1820 0.0393002 0 - 0.957344 2024-04-16T15:48:43.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2895 12 1832 1820 0.0392866 0 - 0.957344 2024-04-16T15:48:44.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2896 12 1832 1820 0.0392731 0 - 0.957344 2024-04-16T15:48:45.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2897 12 1832 1820 0.0392595 0 - 0.957344 2024-04-16T15:48:46.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2898 12 1832 1820 0.039246 0 - 0.957344 2024-04-16T15:48:47.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2899 12 1832 1820 0.0392324 0 - 0.957344 2024-04-16T15:48:48.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:48:48.878920+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:48:48.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:48:48.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2900 12 1832 1820 0.0392189 0 - 0.957344 2024-04-16T15:48:49.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2901 12 1832 1820 0.0392054 0 - 0.957344 2024-04-16T15:48:50.876 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2902 12 1832 1820 0.0391919 0 - 0.957344 2024-04-16T15:48:51.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2903 12 1832 1820 0.0391784 0 - 0.957344 2024-04-16T15:48:52.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2904 12 1832 1820 0.0391649 0 - 0.957344 2024-04-16T15:48:53.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2905 12 1832 1820 0.0391514 0 - 0.957344 2024-04-16T15:48:54.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2906 12 1832 1820 0.0391379 0 - 0.957344 2024-04-16T15:48:55.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2907 12 1832 1820 0.0391245 0 - 0.957344 2024-04-16T15:48:56.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2908 12 1832 1820 0.039111 0 - 0.957344 2024-04-16T15:48:57.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2909 12 1832 1820 0.0390976 0 - 0.957344 2024-04-16T15:48:58.877 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2910 12 1832 1820 0.0390841 0 - 0.957344 2024-04-16T15:48:59.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2911 12 1832 1820 0.0390707 0 - 0.957344 2024-04-16T15:49:00.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2912 12 1832 1820 0.0390573 0 - 0.957344 2024-04-16T15:49:01.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2913 12 1832 1820 0.0390439 0 - 0.957344 2024-04-16T15:49:02.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2914 12 1832 1820 0.0390305 0 - 0.957344 2024-04-16T15:49:03.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2915 12 1832 1820 0.0390171 0 - 0.957344 2024-04-16T15:49:04.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2916 12 1832 1820 0.0390037 0 - 0.957344 2024-04-16T15:49:05.878 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2917 12 1832 1820 0.0389903 0 - 0.957344 2024-04-16T15:49:06.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2918 12 1832 1820 0.038977 0 - 0.957344 2024-04-16T15:49:07.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2919 12 1832 1820 0.0389636 0 - 0.957344 2024-04-16T15:49:08.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:49:08.881531+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:49:08.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:49:08.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2920 12 1832 1820 0.0389503 0 - 0.957344 2024-04-16T15:49:09.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2921 12 1832 1820 0.0389369 0 - 0.957344 2024-04-16T15:49:10.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2922 12 1832 1820 0.0389236 0 - 0.957344 2024-04-16T15:49:11.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2923 12 1832 1820 0.0389103 0 - 0.957344 2024-04-16T15:49:12.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2924 12 1832 1820 0.038897 0 - 0.957344 2024-04-16T15:49:13.879 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2925 12 1832 1820 0.0388837 0 - 0.957344 2024-04-16T15:49:14.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2926 12 1832 1820 0.0388704 0 - 0.957344 2024-04-16T15:49:15.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2927 12 1832 1820 0.0388571 0 - 0.957344 2024-04-16T15:49:16.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2928 12 1832 1820 0.0388439 0 - 0.957344 2024-04-16T15:49:17.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2929 12 1832 1820 0.0388306 0 - 0.957344 2024-04-16T15:49:18.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2930 12 1832 1820 0.0388173 0 - 0.957344 2024-04-16T15:49:19.880 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2931 12 1832 1820 0.0388041 0 - 0.957344 2024-04-16T15:49:20.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2932 12 1832 1820 0.0387909 0 - 0.957344 2024-04-16T15:49:21.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2933 12 1832 1820 0.0387776 0 - 0.957344 2024-04-16T15:49:22.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2934 12 1832 1820 0.0387644 0 - 0.957344 2024-04-16T15:49:23.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2935 12 1832 1820 0.0387512 0 - 0.957344 2024-04-16T15:49:24.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2936 12 1832 1820 0.038738 0 - 0.957344 2024-04-16T15:49:25.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2937 12 1832 1820 0.0387248 0 - 0.957344 2024-04-16T15:49:26.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2938 12 1832 1820 0.0387116 0 - 0.957344 2024-04-16T15:49:27.881 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2939 12 1832 1820 0.0386985 0 - 0.957344 2024-04-16T15:49:28.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:49:28.884215+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:49:28.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:49:28.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2940 12 1832 1820 0.0386853 0 - 0.957344 2024-04-16T15:49:29.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2941 12 1832 1820 0.0386722 0 - 0.957344 2024-04-16T15:49:30.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2942 12 1832 1820 0.038659 0 - 0.957344 2024-04-16T15:49:31.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2943 12 1832 1820 0.0386459 0 - 0.957344 2024-04-16T15:49:32.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2944 12 1832 1820 0.0386328 0 - 0.957344 2024-04-16T15:49:33.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2945 12 1832 1820 0.0386196 0 - 0.957344 2024-04-16T15:49:34.882 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2946 12 1832 1820 0.0386065 0 - 0.957344 2024-04-16T15:49:35.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2947 12 1832 1820 0.0385934 0 - 0.957344 2024-04-16T15:49:36.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2948 12 1832 1820 0.0385803 0 - 0.957344 2024-04-16T15:49:37.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2949 12 1832 1820 0.0385673 0 - 0.957344 2024-04-16T15:49:38.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2950 12 1832 1820 0.0385542 0 - 0.957344 2024-04-16T15:49:39.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2951 12 1832 1820 0.0385411 0 - 0.957344 2024-04-16T15:49:40.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2952 12 1832 1820 0.0385281 0 - 0.957344 2024-04-16T15:49:41.883 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2953 12 1832 1820 0.038515 0 - 0.957344 2024-04-16T15:49:42.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2954 12 1832 1820 0.038502 0 - 0.957344 2024-04-16T15:49:43.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2955 12 1832 1820 0.0384889 0 - 0.957344 2024-04-16T15:49:44.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2956 12 1832 1820 0.0384759 0 - 0.957344 2024-04-16T15:49:45.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2957 12 1832 1820 0.0384629 0 - 0.957344 2024-04-16T15:49:46.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2958 12 1832 1820 0.0384499 0 - 0.957344 2024-04-16T15:49:47.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2959 12 1832 1820 0.0384369 0 - 0.957344 2024-04-16T15:49:48.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:49:48.886921+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:49:48.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:49:48.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2960 12 1832 1820 0.0384239 0 - 0.957344 2024-04-16T15:49:49.884 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2961 12 1832 1820 0.038411 0 - 0.957344 2024-04-16T15:49:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:49:50.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:49:50.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2962 12 1832 1820 0.038398 0 - 0.957344 2024-04-16T15:49:51.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2963 12 1832 1820 0.038385 0 - 0.957344 2024-04-16T15:49:52.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T15:49:52.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:49:52.883+0000 7fbe5eced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:49:52.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2964 12 1832 1820 0.0383721 0 - 0.957344 2024-04-16T15:49:53.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2965 12 1832 1820 0.0383591 0 - 0.957344 2024-04-16T15:49:54.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2966 12 1832 1820 0.0383462 0 - 0.957344 2024-04-16T15:49:55.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2967 12 1832 1820 0.0383333 0 - 0.957344 2024-04-16T15:49:56.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2968 12 1832 1820 0.0383204 0 - 0.957344 2024-04-16T15:49:57.885 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2969 12 1832 1820 0.0383075 0 - 0.957344 2024-04-16T15:49:58.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2970 12 1832 1820 0.0382946 0 - 0.957344 2024-04-16T15:49:59.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2971 12 1832 1820 0.0382817 0 - 0.957344 2024-04-16T15:50:00.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2972 12 1832 1820 0.0382688 0 - 0.957344 2024-04-16T15:50:01.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2973 12 1832 1820 0.0382559 0 - 0.957344 2024-04-16T15:50:02.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2974 12 1832 1820 0.038243 0 - 0.957344 2024-04-16T15:50:03.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2975 12 1832 1820 0.0382302 0 - 0.957344 2024-04-16T15:50:04.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2976 12 1832 1820 0.0382173 0 - 0.957344 2024-04-16T15:50:05.886 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2977 12 1832 1820 0.0382045 0 - 0.957344 2024-04-16T15:50:06.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2978 12 1832 1820 0.0381917 0 - 0.957344 2024-04-16T15:50:07.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2979 12 1832 1820 0.0381789 0 - 0.957344 2024-04-16T15:50:08.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:50:08.889479+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:50:08.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:50:08.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2980 12 1832 1820 0.0381661 0 - 0.957344 2024-04-16T15:50:09.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2981 12 1832 1820 0.0381532 0 - 0.957344 2024-04-16T15:50:10.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2982 12 1832 1820 0.0381405 0 - 0.957344 2024-04-16T15:50:11.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2983 12 1832 1820 0.0381277 0 - 0.957344 2024-04-16T15:50:12.887 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2984 12 1832 1820 0.0381149 0 - 0.957344 2024-04-16T15:50:13.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2985 12 1832 1820 0.0381021 0 - 0.957344 2024-04-16T15:50:14.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2986 12 1832 1820 0.0380894 0 - 0.957344 2024-04-16T15:50:15.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2987 12 1832 1820 0.0380766 0 - 0.957344 2024-04-16T15:50:16.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2988 12 1832 1820 0.0380639 0 - 0.957344 2024-04-16T15:50:17.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2989 12 1832 1820 0.0380511 0 - 0.957344 2024-04-16T15:50:18.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2990 12 1832 1820 0.0380384 0 - 0.957344 2024-04-16T15:50:19.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2991 12 1832 1820 0.0380257 0 - 0.957344 2024-04-16T15:50:20.888 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2992 12 1832 1820 0.038013 0 - 0.957344 2024-04-16T15:50:21.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2993 12 1832 1820 0.0380003 0 - 0.957344 2024-04-16T15:50:22.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2994 12 1832 1820 0.0379876 0 - 0.957344 2024-04-16T15:50:23.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2995 12 1832 1820 0.0379749 0 - 0.957344 2024-04-16T15:50:24.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2996 12 1832 1820 0.0379622 0 - 0.957344 2024-04-16T15:50:25.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2997 12 1832 1820 0.0379496 0 - 0.957344 2024-04-16T15:50:26.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2998 12 1832 1820 0.0379369 0 - 0.957344 2024-04-16T15:50:27.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 2999 12 1832 1820 0.0379243 0 - 0.957344 2024-04-16T15:50:28.889 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:50:28.892111+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:50:28.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:50:28.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3000 12 1832 1820 0.0379116 0 - 0.957344 2024-04-16T15:50:29.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3001 12 1832 1820 0.037899 0 - 0.957344 2024-04-16T15:50:30.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3002 12 1832 1820 0.0378864 0 - 0.957344 2024-04-16T15:50:31.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3003 12 1832 1820 0.0378737 0 - 0.957344 2024-04-16T15:50:32.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3004 12 1832 1820 0.0378611 0 - 0.957344 2024-04-16T15:50:33.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3005 12 1832 1820 0.0378485 0 - 0.957344 2024-04-16T15:50:34.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3006 12 1832 1820 0.0378359 0 - 0.957344 2024-04-16T15:50:35.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3007 12 1832 1820 0.0378234 0 - 0.957344 2024-04-16T15:50:36.890 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3008 12 1832 1820 0.0378108 0 - 0.957344 2024-04-16T15:50:37.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3009 12 1832 1820 0.0377982 0 - 0.957344 2024-04-16T15:50:38.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3010 12 1832 1820 0.0377857 0 - 0.957344 2024-04-16T15:50:39.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3011 12 1832 1820 0.0377731 0 - 0.957344 2024-04-16T15:50:40.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3012 12 1832 1820 0.0377606 0 - 0.957344 2024-04-16T15:50:41.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3013 12 1832 1820 0.037748 0 - 0.957344 2024-04-16T15:50:42.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3014 12 1832 1820 0.0377355 0 - 0.957344 2024-04-16T15:50:43.891 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3015 12 1832 1820 0.037723 0 - 0.957344 2024-04-16T15:50:44.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3016 12 1832 1820 0.0377105 0 - 0.957344 2024-04-16T15:50:45.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3017 12 1832 1820 0.037698 0 - 0.957344 2024-04-16T15:50:46.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3018 12 1832 1820 0.0376855 0 - 0.957344 2024-04-16T15:50:47.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3019 12 1832 1820 0.037673 0 - 0.957344 2024-04-16T15:50:48.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:50:48.894592+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:50:48.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:50:48.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3020 12 1832 1820 0.0376605 0 - 0.957344 2024-04-16T15:50:49.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3021 12 1832 1820 0.0376481 0 - 0.957344 2024-04-16T15:50:50.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3022 12 1832 1820 0.0376356 0 - 0.957344 2024-04-16T15:50:51.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3023 12 1832 1820 0.0376232 0 - 0.957344 2024-04-16T15:50:52.892 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3024 12 1832 1820 0.0376107 0 - 0.957344 2024-04-16T15:50:53.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3025 12 1832 1820 0.0375983 0 - 0.957344 2024-04-16T15:50:54.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3026 12 1832 1820 0.0375859 0 - 0.957344 2024-04-16T15:50:55.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3027 12 1832 1820 0.0375735 0 - 0.957344 2024-04-16T15:50:56.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3028 12 1832 1820 0.037561 0 - 0.957344 2024-04-16T15:50:57.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3029 12 1832 1820 0.0375486 0 - 0.957344 2024-04-16T15:50:58.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3030 12 1832 1820 0.0375363 0 - 0.957344 2024-04-16T15:50:59.893 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3031 12 1832 1820 0.0375239 0 - 0.957344 2024-04-16T15:51:00.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3032 12 1832 1820 0.0375115 0 - 0.957344 2024-04-16T15:51:01.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3033 12 1832 1820 0.0374991 0 - 0.957344 2024-04-16T15:51:02.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3034 12 1832 1820 0.0374868 0 - 0.957344 2024-04-16T15:51:03.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3035 12 1832 1820 0.0374744 0 - 0.957344 2024-04-16T15:51:04.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3036 12 1832 1820 0.0374621 0 - 0.957344 2024-04-16T15:51:05.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3037 12 1832 1820 0.0374497 0 - 0.957344 2024-04-16T15:51:06.894 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3038 12 1832 1820 0.0374374 0 - 0.957344 2024-04-16T15:51:07.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3039 12 1832 1820 0.0374251 0 - 0.957344 2024-04-16T15:51:08.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:51:08.897205+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:51:08.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:51:08.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3040 12 1832 1820 0.0374128 0 - 0.957344 2024-04-16T15:51:09.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3041 12 1832 1820 0.0374005 0 - 0.957344 2024-04-16T15:51:10.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3042 12 1832 1820 0.0373882 0 - 0.957344 2024-04-16T15:51:11.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3043 12 1832 1820 0.0373759 0 - 0.957344 2024-04-16T15:51:12.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3044 12 1832 1820 0.0373636 0 - 0.957344 2024-04-16T15:51:13.166 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-04-16T15:51:13.378 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2c1d76f6d04b413e8bcd12ad16fd1b92 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-04-16T15:51:13.380 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-16T15:51:13.390 INFO:tasks.thrashosds:joining thrashosds 2024-04-16T15:51:13.390 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-16T15:51:13.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3045 12 1832 1820 0.0373513 0 - 0.957344 2024-04-16T15:51:14.895 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3046 12 1832 1820 0.0373391 0 - 0.957344 2024-04-16T15:51:15.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3047 12 1832 1820 0.0373268 0 - 0.957344 2024-04-16T15:51:16.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3048 12 1832 1820 0.0373146 0 - 0.957344 2024-04-16T15:51:17.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3049 12 1832 1820 0.0373023 0 - 0.957344 2024-04-16T15:51:18.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3050 12 1832 1820 0.0372901 0 - 0.957344 2024-04-16T15:51:19.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3051 12 1832 1820 0.0372779 0 - 0.957344 2024-04-16T15:51:20.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3052 12 1832 1820 0.0372657 0 - 0.957344 2024-04-16T15:51:21.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3053 12 1832 1820 0.0372535 0 - 0.957344 2024-04-16T15:51:22.896 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3054 12 1832 1820 0.0372413 0 - 0.957344 2024-04-16T15:51:23.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3055 12 1832 1820 0.0372291 0 - 0.957344 2024-04-16T15:51:24.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3056 12 1832 1820 0.0372169 0 - 0.957344 2024-04-16T15:51:25.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3057 12 1832 1820 0.0372047 0 - 0.957344 2024-04-16T15:51:26.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3058 12 1832 1820 0.0371926 0 - 0.957344 2024-04-16T15:51:27.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3059 12 1832 1820 0.0371804 0 - 0.957344 2024-04-16T15:51:28.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:51:28.899784+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:51:28.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:51:28.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3060 12 1832 1820 0.0371683 0 - 0.957344 2024-04-16T15:51:29.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3061 12 1832 1820 0.0371561 0 - 0.957344 2024-04-16T15:51:30.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3062 12 1832 1820 0.037144 0 - 0.957344 2024-04-16T15:51:31.897 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3063 12 1832 1820 0.0371318 0 - 0.957344 2024-04-16T15:51:32.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3064 12 1832 1820 0.0371197 0 - 0.957344 2024-04-16T15:51:33.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3065 12 1832 1820 0.0371076 0 - 0.957344 2024-04-16T15:51:34.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3066 12 1832 1820 0.0370955 0 - 0.957344 2024-04-16T15:51:35.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3067 12 1832 1820 0.0370834 0 - 0.957344 2024-04-16T15:51:36.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3068 12 1832 1820 0.0370713 0 - 0.957344 2024-04-16T15:51:37.898 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3069 12 1832 1820 0.0370593 0 - 0.957344 2024-04-16T15:51:38.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3070 12 1832 1820 0.0370472 0 - 0.957344 2024-04-16T15:51:39.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3071 12 1832 1820 0.0370351 0 - 0.957344 2024-04-16T15:51:40.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3072 12 1832 1820 0.0370231 0 - 0.957344 2024-04-16T15:51:41.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3073 12 1832 1820 0.037011 0 - 0.957344 2024-04-16T15:51:42.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3074 12 1832 1820 0.036999 0 - 0.957344 2024-04-16T15:51:43.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3075 12 1832 1820 0.0369869 0 - 0.957344 2024-04-16T15:51:44.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3076 12 1832 1820 0.0369749 0 - 0.957344 2024-04-16T15:51:45.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3077 12 1832 1820 0.0369629 0 - 0.957344 2024-04-16T15:51:46.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3078 12 1832 1820 0.0369509 0 - 0.957344 2024-04-16T15:51:47.899 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3079 12 1832 1820 0.0369389 0 - 0.957344 2024-04-16T15:51:48.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:51:48.902252+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:51:48.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:51:48.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3080 12 1832 1820 0.0369269 0 - 0.957344 2024-04-16T15:51:49.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3081 12 1832 1820 0.0369149 0 - 0.957344 2024-04-16T15:51:50.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3082 12 1832 1820 0.0369029 0 - 0.957344 2024-04-16T15:51:51.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3083 12 1832 1820 0.036891 0 - 0.957344 2024-04-16T15:51:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:51:52.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T15:51:52.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3084 12 1832 1820 0.036879 0 - 0.957344 2024-04-16T15:51:53.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3085 12 1832 1820 0.0368671 0 - 0.957344 2024-04-16T15:51:54.803 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T15:51:54.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T15:51:54.900 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3086 12 1832 1820 0.0368551 0 - 0.957344 2024-04-16T15:51:54.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:51:54.926+0000 7f545039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:51:55.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3087 12 1832 1820 0.0368432 0 - 0.957344 2024-04-16T15:51:56.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3088 12 1832 1820 0.0368312 0 - 0.957344 2024-04-16T15:51:57.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3089 12 1832 1820 0.0368193 0 - 0.957344 2024-04-16T15:51:58.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3090 12 1832 1820 0.0368074 0 - 0.957344 2024-04-16T15:51:59.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3091 12 1832 1820 0.0367955 0 - 0.957344 2024-04-16T15:52:00.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3092 12 1832 1820 0.0367836 0 - 0.957344 2024-04-16T15:52:01.901 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3093 12 1832 1820 0.0367717 0 - 0.957344 2024-04-16T15:52:02.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3094 12 1832 1820 0.0367598 0 - 0.957344 2024-04-16T15:52:03.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3095 12 1832 1820 0.0367479 0 - 0.957344 2024-04-16T15:52:04.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3096 12 1832 1820 0.0367361 0 - 0.957344 2024-04-16T15:52:05.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3097 12 1832 1820 0.0367242 0 - 0.957344 2024-04-16T15:52:06.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3098 12 1832 1820 0.0367123 0 - 0.957344 2024-04-16T15:52:07.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3099 12 1832 1820 0.0367005 0 - 0.957344 2024-04-16T15:52:08.902 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:52:08.904997+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:52:08.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:52:08.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3100 12 1832 1820 0.0366887 0 - 0.957344 2024-04-16T15:52:09.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3101 12 1832 1820 0.0366768 0 - 0.957344 2024-04-16T15:52:10.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3102 12 1832 1820 0.036665 0 - 0.957344 2024-04-16T15:52:11.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3103 12 1832 1820 0.0366532 0 - 0.957344 2024-04-16T15:52:12.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3104 12 1832 1820 0.0366414 0 - 0.957344 2024-04-16T15:52:13.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3105 12 1832 1820 0.0366296 0 - 0.957344 2024-04-16T15:52:14.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3106 12 1832 1820 0.0366178 0 - 0.957344 2024-04-16T15:52:15.903 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3107 12 1832 1820 0.036606 0 - 0.957344 2024-04-16T15:52:16.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3108 12 1832 1820 0.0365942 0 - 0.957344 2024-04-16T15:52:17.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3109 12 1832 1820 0.0365825 0 - 0.957344 2024-04-16T15:52:18.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3110 12 1832 1820 0.0365707 0 - 0.957344 2024-04-16T15:52:19.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3111 12 1832 1820 0.0365589 0 - 0.957344 2024-04-16T15:52:20.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3112 12 1832 1820 0.0365472 0 - 0.957344 2024-04-16T15:52:21.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3113 12 1832 1820 0.0365354 0 - 0.957344 2024-04-16T15:52:22.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3114 12 1832 1820 0.0365237 0 - 0.957344 2024-04-16T15:52:23.904 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3115 12 1832 1820 0.036512 0 - 0.957344 2024-04-16T15:52:24.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3116 12 1832 1820 0.0365003 0 - 0.957344 2024-04-16T15:52:25.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3117 12 1832 1820 0.0364886 0 - 0.957344 2024-04-16T15:52:26.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3118 12 1832 1820 0.0364769 0 - 0.957344 2024-04-16T15:52:27.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3119 12 1832 1820 0.0364652 0 - 0.957344 2024-04-16T15:52:28.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:52:28.907653+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:52:28.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:52:28.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3120 12 1832 1820 0.0364535 0 - 0.957344 2024-04-16T15:52:29.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3121 12 1832 1820 0.0364418 0 - 0.957344 2024-04-16T15:52:30.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3122 12 1832 1820 0.0364301 0 - 0.957344 2024-04-16T15:52:31.905 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3123 12 1832 1820 0.0364185 0 - 0.957344 2024-04-16T15:52:32.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3124 12 1832 1820 0.0364068 0 - 0.957344 2024-04-16T15:52:33.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3125 12 1832 1820 0.0363952 0 - 0.957344 2024-04-16T15:52:34.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3126 12 1832 1820 0.0363835 0 - 0.957344 2024-04-16T15:52:35.907 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3127 12 1832 1820 0.0363719 0 - 0.957344 2024-04-16T15:52:36.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3128 12 1832 1820 0.0363602 0 - 0.957344 2024-04-16T15:52:37.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3129 12 1832 1820 0.0363486 0 - 0.957344 2024-04-16T15:52:38.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3130 12 1832 1820 0.036337 0 - 0.957344 2024-04-16T15:52:39.906 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3131 12 1832 1820 0.0363254 0 - 0.957344 2024-04-16T15:52:40.907 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3132 12 1832 1820 0.0363138 0 - 0.957344 2024-04-16T15:52:41.907 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3133 12 1832 1820 0.0363022 0 - 0.957344 2024-04-16T15:52:42.907 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3134 12 1832 1820 0.0362906 0 - 0.957344 2024-04-16T15:52:43.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3135 12 1832 1820 0.0362791 0 - 0.957344 2024-04-16T15:52:44.907 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3136 12 1832 1820 0.0362675 0 - 0.957344 2024-04-16T15:52:45.907 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3137 12 1832 1820 0.0362559 0 - 0.957344 2024-04-16T15:52:46.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3138 12 1832 1820 0.0362444 0 - 0.957344 2024-04-16T15:52:47.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3139 12 1832 1820 0.0362328 0 - 0.957344 2024-04-16T15:52:48.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:52:48.910287+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:52:48.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:52:48.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3140 12 1832 1820 0.0362213 0 - 0.957344 2024-04-16T15:52:49.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3141 12 1832 1820 0.0362098 0 - 0.957344 2024-04-16T15:52:50.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3142 12 1832 1820 0.0361982 0 - 0.957344 2024-04-16T15:52:51.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3143 12 1832 1820 0.0361867 0 - 0.957344 2024-04-16T15:52:52.908 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3144 12 1832 1820 0.0361752 0 - 0.957344 2024-04-16T15:52:53.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3145 12 1832 1820 0.0361637 0 - 0.957344 2024-04-16T15:52:54.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3146 12 1832 1820 0.0361522 0 - 0.957344 2024-04-16T15:52:55.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3147 12 1832 1820 0.0361407 0 - 0.957344 2024-04-16T15:52:56.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3148 12 1832 1820 0.0361292 0 - 0.957344 2024-04-16T15:52:57.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3149 12 1832 1820 0.0361178 0 - 0.957344 2024-04-16T15:52:58.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3150 12 1832 1820 0.0361063 0 - 0.957344 2024-04-16T15:52:59.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3151 12 1832 1820 0.0360948 0 - 0.957344 2024-04-16T15:53:00.909 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3152 12 1832 1820 0.0360834 0 - 0.957344 2024-04-16T15:53:01.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3153 12 1832 1820 0.0360719 0 - 0.957344 2024-04-16T15:53:02.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3154 12 1832 1820 0.0360605 0 - 0.957344 2024-04-16T15:53:03.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3155 12 1832 1820 0.0360491 0 - 0.957344 2024-04-16T15:53:04.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3156 12 1832 1820 0.0360377 0 - 0.957344 2024-04-16T15:53:05.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3157 12 1832 1820 0.0360262 0 - 0.957344 2024-04-16T15:53:06.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3158 12 1832 1820 0.0360148 0 - 0.957344 2024-04-16T15:53:07.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3159 12 1832 1820 0.0360034 0 - 0.957344 2024-04-16T15:53:08.910 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:53:08.912988+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:53:08.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:53:08.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3160 12 1832 1820 0.035992 0 - 0.957344 2024-04-16T15:53:09.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3161 12 1832 1820 0.0359807 0 - 0.957344 2024-04-16T15:53:10.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3162 12 1832 1820 0.0359693 0 - 0.957344 2024-04-16T15:53:11.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3163 12 1832 1820 0.0359579 0 - 0.957344 2024-04-16T15:53:12.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3164 12 1832 1820 0.0359465 0 - 0.957344 2024-04-16T15:53:13.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3165 12 1832 1820 0.0359352 0 - 0.957344 2024-04-16T15:53:14.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3166 12 1832 1820 0.0359238 0 - 0.957344 2024-04-16T15:53:15.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3167 12 1832 1820 0.0359125 0 - 0.957344 2024-04-16T15:53:16.911 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3168 12 1832 1820 0.0359012 0 - 0.957344 2024-04-16T15:53:17.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3169 12 1832 1820 0.0358898 0 - 0.957344 2024-04-16T15:53:18.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3170 12 1832 1820 0.0358785 0 - 0.957344 2024-04-16T15:53:19.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3171 12 1832 1820 0.0358672 0 - 0.957344 2024-04-16T15:53:20.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3172 12 1832 1820 0.0358559 0 - 0.957344 2024-04-16T15:53:21.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3173 12 1832 1820 0.0358446 0 - 0.957344 2024-04-16T15:53:22.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3174 12 1832 1820 0.0358333 0 - 0.957344 2024-04-16T15:53:23.912 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3175 12 1832 1820 0.035822 0 - 0.957344 2024-04-16T15:53:24.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3176 12 1832 1820 0.0358107 0 - 0.957344 2024-04-16T15:53:25.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3177 12 1832 1820 0.0357994 0 - 0.957344 2024-04-16T15:53:26.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3178 12 1832 1820 0.0357882 0 - 0.957344 2024-04-16T15:53:27.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3179 12 1832 1820 0.0357769 0 - 0.957344 2024-04-16T15:53:28.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:53:28.915682+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:53:28.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:53:28.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3180 12 1832 1820 0.0357657 0 - 0.957344 2024-04-16T15:53:29.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3181 12 1832 1820 0.0357544 0 - 0.957344 2024-04-16T15:53:30.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3182 12 1832 1820 0.0357432 0 - 0.957344 2024-04-16T15:53:31.913 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3183 12 1832 1820 0.035732 0 - 0.957344 2024-04-16T15:53:32.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3184 12 1832 1820 0.0357207 0 - 0.957344 2024-04-16T15:53:33.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3185 12 1832 1820 0.0357095 0 - 0.957344 2024-04-16T15:53:34.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3186 12 1832 1820 0.0356983 0 - 0.957344 2024-04-16T15:53:35.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3187 12 1832 1820 0.0356871 0 - 0.957344 2024-04-16T15:53:36.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3188 12 1832 1820 0.0356759 0 - 0.957344 2024-04-16T15:53:37.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3189 12 1832 1820 0.0356647 0 - 0.957344 2024-04-16T15:53:38.914 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3190 12 1832 1820 0.0356536 0 - 0.957344 2024-04-16T15:53:39.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3191 12 1832 1820 0.0356424 0 - 0.957344 2024-04-16T15:53:40.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3192 12 1832 1820 0.0356312 0 - 0.957344 2024-04-16T15:53:41.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3193 12 1832 1820 0.0356201 0 - 0.957344 2024-04-16T15:53:42.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3194 12 1832 1820 0.0356089 0 - 0.957344 2024-04-16T15:53:43.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3195 12 1832 1820 0.0355978 0 - 0.957344 2024-04-16T15:53:44.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3196 12 1832 1820 0.0355866 0 - 0.957344 2024-04-16T15:53:45.915 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3197 12 1832 1820 0.0355755 0 - 0.957344 2024-04-16T15:53:46.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3198 12 1832 1820 0.0355644 0 - 0.957344 2024-04-16T15:53:47.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3199 12 1832 1820 0.0355532 0 - 0.957344 2024-04-16T15:53:48.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:53:48.918419+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:53:48.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:53:48.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3200 12 1832 1820 0.0355421 0 - 0.957344 2024-04-16T15:53:49.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3201 12 1832 1820 0.035531 0 - 0.957344 2024-04-16T15:53:50.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3202 12 1832 1820 0.0355199 0 - 0.957344 2024-04-16T15:53:51.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3203 12 1832 1820 0.0355089 0 - 0.957344 2024-04-16T15:53:52.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3204 12 1832 1820 0.0354978 0 - 0.957344 2024-04-16T15:53:53.916 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3205 12 1832 1820 0.0354867 0 - 0.957344 2024-04-16T15:53:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:53:54.844 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T15:53:54.859 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T15:53:54.868 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T15:53:54.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T15:53:54.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3206 12 1832 1820 0.0354756 0 - 0.957344 2024-04-16T15:53:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-16T15:53:54.995+0000 7fb32106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T15:53:55.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3207 12 1832 1820 0.0354646 0 - 0.957344 2024-04-16T15:53:56.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3208 12 1832 1820 0.0354535 0 - 0.957344 2024-04-16T15:53:57.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3209 12 1832 1820 0.0354425 0 - 0.957344 2024-04-16T15:53:58.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3210 12 1832 1820 0.0354314 0 - 0.957344 2024-04-16T15:53:59.917 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3211 12 1832 1820 0.0354204 0 - 0.957344 2024-04-16T15:54:00.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3212 12 1832 1820 0.0354094 0 - 0.957344 2024-04-16T15:54:01.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3213 12 1832 1820 0.0353983 0 - 0.957344 2024-04-16T15:54:02.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3214 12 1832 1820 0.0353873 0 - 0.957344 2024-04-16T15:54:03.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3215 12 1832 1820 0.0353763 0 - 0.957344 2024-04-16T15:54:04.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3216 12 1832 1820 0.0353653 0 - 0.957344 2024-04-16T15:54:05.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3217 12 1832 1820 0.0353543 0 - 0.957344 2024-04-16T15:54:06.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3218 12 1832 1820 0.0353433 0 - 0.957344 2024-04-16T15:54:07.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3219 12 1832 1820 0.0353324 0 - 0.957344 2024-04-16T15:54:08.918 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:54:08.920900+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:54:08.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:54:08.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3220 12 1832 1820 0.0353214 0 - 0.957344 2024-04-16T15:54:09.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3221 12 1832 1820 0.0353104 0 - 0.957344 2024-04-16T15:54:10.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3222 12 1832 1820 0.0352995 0 - 0.957344 2024-04-16T15:54:11.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3223 12 1832 1820 0.0352885 0 - 0.957344 2024-04-16T15:54:12.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3224 12 1832 1820 0.0352776 0 - 0.957344 2024-04-16T15:54:13.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3225 12 1832 1820 0.0352666 0 - 0.957344 2024-04-16T15:54:14.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3226 12 1832 1820 0.0352557 0 - 0.957344 2024-04-16T15:54:15.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3227 12 1832 1820 0.0352448 0 - 0.957344 2024-04-16T15:54:16.919 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3228 12 1832 1820 0.0352338 0 - 0.957344 2024-04-16T15:54:17.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3229 12 1832 1820 0.0352229 0 - 0.957344 2024-04-16T15:54:18.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3230 12 1832 1820 0.035212 0 - 0.957344 2024-04-16T15:54:19.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3231 12 1832 1820 0.0352011 0 - 0.957344 2024-04-16T15:54:20.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3232 12 1832 1820 0.0351902 0 - 0.957344 2024-04-16T15:54:21.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3233 12 1832 1820 0.0351794 0 - 0.957344 2024-04-16T15:54:22.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3234 12 1832 1820 0.0351685 0 - 0.957344 2024-04-16T15:54:23.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3235 12 1832 1820 0.0351576 0 - 0.957344 2024-04-16T15:54:24.920 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3236 12 1832 1820 0.0351467 0 - 0.957344 2024-04-16T15:54:25.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3237 12 1832 1820 0.0351359 0 - 0.957344 2024-04-16T15:54:26.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3238 12 1832 1820 0.035125 0 - 0.957344 2024-04-16T15:54:27.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3239 12 1832 1820 0.0351142 0 - 0.957344 2024-04-16T15:54:28.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:54:28.923584+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:54:28.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:54:28.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3240 12 1832 1820 0.0351033 0 - 0.957344 2024-04-16T15:54:29.922 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3241 12 1832 1820 0.0350925 0 - 0.957344 2024-04-16T15:54:30.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3242 12 1832 1820 0.0350817 0 - 0.957344 2024-04-16T15:54:31.921 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3243 12 1832 1820 0.0350709 0 - 0.957344 2024-04-16T15:54:32.922 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3244 12 1832 1820 0.0350601 0 - 0.957344 2024-04-16T15:54:33.922 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3245 12 1832 1820 0.0350493 0 - 0.957344 2024-04-16T15:54:34.922 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3246 12 1832 1820 0.0350385 0 - 0.957344 2024-04-16T15:54:35.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3247 12 1832 1820 0.0350277 0 - 0.957344 2024-04-16T15:54:36.922 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3248 12 1832 1820 0.0350169 0 - 0.957344 2024-04-16T15:54:37.922 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3249 12 1832 1820 0.0350061 0 - 0.957344 2024-04-16T15:54:38.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3250 12 1832 1820 0.0349953 0 - 0.957344 2024-04-16T15:54:39.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3251 12 1832 1820 0.0349846 0 - 0.957344 2024-04-16T15:54:40.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3252 12 1832 1820 0.0349738 0 - 0.957344 2024-04-16T15:54:41.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3253 12 1832 1820 0.0349631 0 - 0.957344 2024-04-16T15:54:42.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3254 12 1832 1820 0.0349523 0 - 0.957344 2024-04-16T15:54:43.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3255 12 1832 1820 0.0349416 0 - 0.957344 2024-04-16T15:54:44.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3256 12 1832 1820 0.0349309 0 - 0.957344 2024-04-16T15:54:45.923 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3257 12 1832 1820 0.0349201 0 - 0.957344 2024-04-16T15:54:46.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3258 12 1832 1820 0.0349094 0 - 0.957344 2024-04-16T15:54:47.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3259 12 1832 1820 0.0348987 0 - 0.957344 2024-04-16T15:54:48.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:54:48.926369+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:54:48.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:54:48.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3260 12 1832 1820 0.034888 0 - 0.957344 2024-04-16T15:54:49.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3261 12 1832 1820 0.0348773 0 - 0.957344 2024-04-16T15:54:50.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3262 12 1832 1820 0.0348666 0 - 0.957344 2024-04-16T15:54:51.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3263 12 1832 1820 0.0348559 0 - 0.957344 2024-04-16T15:54:52.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3264 12 1832 1820 0.0348452 0 - 0.957344 2024-04-16T15:54:53.924 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3265 12 1832 1820 0.0348346 0 - 0.957344 2024-04-16T15:54:54.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3266 12 1832 1820 0.0348239 0 - 0.957344 2024-04-16T15:54:55.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3267 12 1832 1820 0.0348132 0 - 0.957344 2024-04-16T15:54:56.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3268 12 1832 1820 0.0348026 0 - 0.957344 2024-04-16T15:54:57.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3269 12 1832 1820 0.0347919 0 - 0.957344 2024-04-16T15:54:58.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3270 12 1832 1820 0.0347813 0 - 0.957344 2024-04-16T15:54:59.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3271 12 1832 1820 0.0347707 0 - 0.957344 2024-04-16T15:55:00.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3272 12 1832 1820 0.03476 0 - 0.957344 2024-04-16T15:55:01.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3273 12 1832 1820 0.0347494 0 - 0.957344 2024-04-16T15:55:02.925 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3274 12 1832 1820 0.0347388 0 - 0.957344 2024-04-16T15:55:03.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3275 12 1832 1820 0.0347282 0 - 0.957344 2024-04-16T15:55:04.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3276 12 1832 1820 0.0347176 0 - 0.957344 2024-04-16T15:55:05.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3277 12 1832 1820 0.034707 0 - 0.957344 2024-04-16T15:55:06.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3278 12 1832 1820 0.0346964 0 - 0.957344 2024-04-16T15:55:07.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3279 12 1832 1820 0.0346858 0 - 0.957344 2024-04-16T15:55:08.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:55:08.928895+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:55:08.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:55:08.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3280 12 1832 1820 0.0346753 0 - 0.957344 2024-04-16T15:55:09.926 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3281 12 1832 1820 0.0346647 0 - 0.957344 2024-04-16T15:55:10.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3282 12 1832 1820 0.0346541 0 - 0.957344 2024-04-16T15:55:11.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3283 12 1832 1820 0.0346436 0 - 0.957344 2024-04-16T15:55:12.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3284 12 1832 1820 0.034633 0 - 0.957344 2024-04-16T15:55:13.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3285 12 1832 1820 0.0346225 0 - 0.957344 2024-04-16T15:55:14.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3286 12 1832 1820 0.0346119 0 - 0.957344 2024-04-16T15:55:15.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3287 12 1832 1820 0.0346014 0 - 0.957344 2024-04-16T15:55:16.927 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3288 12 1832 1820 0.0345909 0 - 0.957344 2024-04-16T15:55:17.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3289 12 1832 1820 0.0345804 0 - 0.957344 2024-04-16T15:55:18.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3290 12 1832 1820 0.0345699 0 - 0.957344 2024-04-16T15:55:19.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3291 12 1832 1820 0.0345594 0 - 0.957344 2024-04-16T15:55:20.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3292 12 1832 1820 0.0345489 0 - 0.957344 2024-04-16T15:55:21.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3293 12 1832 1820 0.0345384 0 - 0.957344 2024-04-16T15:55:22.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3294 12 1832 1820 0.0345279 0 - 0.957344 2024-04-16T15:55:23.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3295 12 1832 1820 0.0345174 0 - 0.957344 2024-04-16T15:55:24.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3296 12 1832 1820 0.0345069 0 - 0.957344 2024-04-16T15:55:25.928 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3297 12 1832 1820 0.0344965 0 - 0.957344 2024-04-16T15:55:26.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3298 12 1832 1820 0.034486 0 - 0.957344 2024-04-16T15:55:27.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3299 12 1832 1820 0.0344756 0 - 0.957344 2024-04-16T15:55:28.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:55:28.931541+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:55:28.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:55:28.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3300 12 1832 1820 0.0344651 0 - 0.957344 2024-04-16T15:55:29.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3301 12 1832 1820 0.0344547 0 - 0.957344 2024-04-16T15:55:30.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3302 12 1832 1820 0.0344442 0 - 0.957344 2024-04-16T15:55:31.929 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3303 12 1832 1820 0.0344338 0 - 0.957344 2024-04-16T15:55:32.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3304 12 1832 1820 0.0344234 0 - 0.957344 2024-04-16T15:55:33.930 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3305 12 1832 1820 0.034413 0 - 0.957344 2024-04-16T15:55:34.930 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3306 12 1832 1820 0.0344026 0 - 0.957344 2024-04-16T15:55:35.930 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3307 12 1832 1820 0.0343922 0 - 0.957344 2024-04-16T15:55:36.930 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3308 12 1832 1820 0.0343818 0 - 0.957344 2024-04-16T15:55:37.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3309 12 1832 1820 0.0343714 0 - 0.957344 2024-04-16T15:55:38.930 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3310 12 1832 1820 0.034361 0 - 0.957344 2024-04-16T15:55:39.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3311 12 1832 1820 0.0343506 0 - 0.957344 2024-04-16T15:55:40.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3312 12 1832 1820 0.0343402 0 - 0.957344 2024-04-16T15:55:41.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3313 12 1832 1820 0.0343299 0 - 0.957344 2024-04-16T15:55:42.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3314 12 1832 1820 0.0343195 0 - 0.957344 2024-04-16T15:55:43.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3315 12 1832 1820 0.0343092 0 - 0.957344 2024-04-16T15:55:44.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3316 12 1832 1820 0.0342988 0 - 0.957344 2024-04-16T15:55:45.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3317 12 1832 1820 0.0342885 0 - 0.957344 2024-04-16T15:55:46.931 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3318 12 1832 1820 0.0342781 0 - 0.957344 2024-04-16T15:55:47.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3319 12 1832 1820 0.0342678 0 - 0.957344 2024-04-16T15:55:48.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:55:48.934289+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:55:48.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:55:48.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3320 12 1832 1820 0.0342575 0 - 0.957344 2024-04-16T15:55:49.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3321 12 1832 1820 0.0342472 0 - 0.957344 2024-04-16T15:55:50.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3322 12 1832 1820 0.0342369 0 - 0.957344 2024-04-16T15:55:51.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3323 12 1832 1820 0.0342266 0 - 0.957344 2024-04-16T15:55:52.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3324 12 1832 1820 0.0342163 0 - 0.957344 2024-04-16T15:55:53.932 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3325 12 1832 1820 0.034206 0 - 0.957344 2024-04-16T15:55:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T15:55:54.912 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T15:55:54.932 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T15:55:54.932 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T15:55:54.932 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T15:55:54.932 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:55:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:55:54.936 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-04-16T15:55:54.955 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T15:55:54.955 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T15:55:54.955 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:55:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:55:54.956 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-16T15:55:54.974 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T15:55:54.974 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T15:55:54.974 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:55:54.977 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3326 12 1832 1820 0.0341957 0 - 0.957344 2024-04-16T15:55:55.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3327 12 1832 1820 0.0341854 0 - 0.957344 2024-04-16T15:55:56.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3328 12 1832 1820 0.0341751 0 - 0.957344 2024-04-16T15:55:57.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3329 12 1832 1820 0.0341649 0 - 0.957344 2024-04-16T15:55:58.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3330 12 1832 1820 0.0341546 0 - 0.957344 2024-04-16T15:55:59.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3331 12 1832 1820 0.0341444 0 - 0.957344 2024-04-16T15:56:00.933 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3332 12 1832 1820 0.0341341 0 - 0.957344 2024-04-16T15:56:01.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3333 12 1832 1820 0.0341239 0 - 0.957344 2024-04-16T15:56:02.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3334 12 1832 1820 0.0341136 0 - 0.957344 2024-04-16T15:56:03.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3335 12 1832 1820 0.0341034 0 - 0.957344 2024-04-16T15:56:04.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3336 12 1832 1820 0.0340932 0 - 0.957344 2024-04-16T15:56:05.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3337 12 1832 1820 0.034083 0 - 0.957344 2024-04-16T15:56:06.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3338 12 1832 1820 0.0340728 0 - 0.957344 2024-04-16T15:56:07.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3339 12 1832 1820 0.0340625 0 - 0.957344 2024-04-16T15:56:08.934 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:56:08.936995+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:56:08.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:56:08.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3340 12 1832 1820 0.0340524 0 - 0.957344 2024-04-16T15:56:09.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3341 12 1832 1820 0.0340422 0 - 0.957344 2024-04-16T15:56:10.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3342 12 1832 1820 0.034032 0 - 0.957344 2024-04-16T15:56:11.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3343 12 1832 1820 0.0340218 0 - 0.957344 2024-04-16T15:56:12.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3344 12 1832 1820 0.0340116 0 - 0.957344 2024-04-16T15:56:13.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3345 12 1832 1820 0.0340014 0 - 0.957344 2024-04-16T15:56:14.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3346 12 1832 1820 0.0339913 0 - 0.957344 2024-04-16T15:56:15.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3347 12 1832 1820 0.0339811 0 - 0.957344 2024-04-16T15:56:16.935 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3348 12 1832 1820 0.033971 0 - 0.957344 2024-04-16T15:56:17.936 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3349 12 1832 1820 0.0339608 0 - 0.957344 2024-04-16T15:56:18.936 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3350 12 1832 1820 0.0339507 0 - 0.957344 2024-04-16T15:56:19.936 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3351 12 1832 1820 0.0339406 0 - 0.957344 2024-04-16T15:56:20.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3352 12 1832 1820 0.0339304 0 - 0.957344 2024-04-16T15:56:21.936 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3353 12 1832 1820 0.0339203 0 - 0.957344 2024-04-16T15:56:22.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3354 12 1832 1820 0.0339102 0 - 0.957344 2024-04-16T15:56:23.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3355 12 1832 1820 0.0339001 0 - 0.957344 2024-04-16T15:56:24.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3356 12 1832 1820 0.03389 0 - 0.957344 2024-04-16T15:56:25.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3357 12 1832 1820 0.0338799 0 - 0.957344 2024-04-16T15:56:26.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3358 12 1832 1820 0.0338698 0 - 0.957344 2024-04-16T15:56:27.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3359 12 1832 1820 0.0338597 0 - 0.957344 2024-04-16T15:56:28.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:56:28.939852+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:56:28.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:56:28.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3360 12 1832 1820 0.0338497 0 - 0.957344 2024-04-16T15:56:29.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3361 12 1832 1820 0.0338396 0 - 0.957344 2024-04-16T15:56:30.937 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3362 12 1832 1820 0.0338295 0 - 0.957344 2024-04-16T15:56:31.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3363 12 1832 1820 0.0338195 0 - 0.957344 2024-04-16T15:56:32.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3364 12 1832 1820 0.0338094 0 - 0.957344 2024-04-16T15:56:33.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3365 12 1832 1820 0.0337994 0 - 0.957344 2024-04-16T15:56:34.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3366 12 1832 1820 0.0337893 0 - 0.957344 2024-04-16T15:56:35.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3367 12 1832 1820 0.0337793 0 - 0.957344 2024-04-16T15:56:36.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3368 12 1832 1820 0.0337693 0 - 0.957344 2024-04-16T15:56:37.938 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3369 12 1832 1820 0.0337592 0 - 0.957344 2024-04-16T15:56:38.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3370 12 1832 1820 0.0337492 0 - 0.957344 2024-04-16T15:56:39.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3371 12 1832 1820 0.0337392 0 - 0.957344 2024-04-16T15:56:40.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3372 12 1832 1820 0.0337292 0 - 0.957344 2024-04-16T15:56:41.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3373 12 1832 1820 0.0337192 0 - 0.957344 2024-04-16T15:56:42.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3374 12 1832 1820 0.0337092 0 - 0.957344 2024-04-16T15:56:43.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3375 12 1832 1820 0.0336992 0 - 0.957344 2024-04-16T15:56:44.939 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3376 12 1832 1820 0.0336892 0 - 0.957344 2024-04-16T15:56:45.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3377 12 1832 1820 0.0336793 0 - 0.957344 2024-04-16T15:56:46.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3378 12 1832 1820 0.0336693 0 - 0.957344 2024-04-16T15:56:47.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3379 12 1832 1820 0.0336593 0 - 0.957344 2024-04-16T15:56:48.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:56:48.942505+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:56:48.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:56:48.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3380 12 1832 1820 0.0336494 0 - 0.957344 2024-04-16T15:56:49.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3381 12 1832 1820 0.0336394 0 - 0.957344 2024-04-16T15:56:50.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3382 12 1832 1820 0.0336295 0 - 0.957344 2024-04-16T15:56:51.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3383 12 1832 1820 0.0336195 0 - 0.957344 2024-04-16T15:56:52.940 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3384 12 1832 1820 0.0336096 0 - 0.957344 2024-04-16T15:56:53.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3385 12 1832 1820 0.0335997 0 - 0.957344 2024-04-16T15:56:54.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3386 12 1832 1820 0.0335897 0 - 0.957344 2024-04-16T15:56:55.076 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None, False)> 2024-04-16T15:56:55.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3387 12 1832 1820 0.0335798 0 - 0.957344 2024-04-16T15:56:56.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3388 12 1832 1820 0.0335699 0 - 0.957344 2024-04-16T15:56:57.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3389 12 1832 1820 0.03356 0 - 0.957344 2024-04-16T15:56:58.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3390 12 1832 1820 0.0335501 0 - 0.957344 2024-04-16T15:56:59.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3391 12 1832 1820 0.0335402 0 - 0.957344 2024-04-16T15:57:00.941 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3392 12 1832 1820 0.0335303 0 - 0.957344 2024-04-16T15:57:01.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3393 12 1832 1820 0.0335204 0 - 0.957344 2024-04-16T15:57:02.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3394 12 1832 1820 0.0335106 0 - 0.957344 2024-04-16T15:57:03.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3395 12 1832 1820 0.0335007 0 - 0.957344 2024-04-16T15:57:04.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3396 12 1832 1820 0.0334908 0 - 0.957344 2024-04-16T15:57:05.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3397 12 1832 1820 0.033481 0 - 0.957344 2024-04-16T15:57:06.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3398 12 1832 1820 0.0334711 0 - 0.957344 2024-04-16T15:57:07.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3399 12 1832 1820 0.0334613 0 - 0.957344 2024-04-16T15:57:08.942 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:57:08.945126+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:57:08.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:57:08.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3400 12 1832 1820 0.0334514 0 - 0.957344 2024-04-16T15:57:09.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3401 12 1832 1820 0.0334416 0 - 0.957344 2024-04-16T15:57:10.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3402 12 1832 1820 0.0334318 0 - 0.957344 2024-04-16T15:57:11.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3403 12 1832 1820 0.0334219 0 - 0.957344 2024-04-16T15:57:12.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3404 12 1832 1820 0.0334121 0 - 0.957344 2024-04-16T15:57:13.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3405 12 1832 1820 0.0334023 0 - 0.957344 2024-04-16T15:57:14.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3406 12 1832 1820 0.0333925 0 - 0.957344 2024-04-16T15:57:15.943 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3407 12 1832 1820 0.0333827 0 - 0.957344 2024-04-16T15:57:16.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3408 12 1832 1820 0.0333729 0 - 0.957344 2024-04-16T15:57:17.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3409 12 1832 1820 0.0333631 0 - 0.957344 2024-04-16T15:57:18.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3410 12 1832 1820 0.0333533 0 - 0.957344 2024-04-16T15:57:19.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3411 12 1832 1820 0.0333435 0 - 0.957344 2024-04-16T15:57:20.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3412 12 1832 1820 0.0333338 0 - 0.957344 2024-04-16T15:57:21.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3413 12 1832 1820 0.033324 0 - 0.957344 2024-04-16T15:57:22.944 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3414 12 1832 1820 0.0333142 0 - 0.957344 2024-04-16T15:57:23.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3415 12 1832 1820 0.0333045 0 - 0.957344 2024-04-16T15:57:24.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3416 12 1832 1820 0.0332947 0 - 0.957344 2024-04-16T15:57:25.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3417 12 1832 1820 0.033285 0 - 0.957344 2024-04-16T15:57:26.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3418 12 1832 1820 0.0332753 0 - 0.957344 2024-04-16T15:57:27.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3419 12 1832 1820 0.0332655 0 - 0.957344 2024-04-16T15:57:28.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:57:28.947837+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:57:28.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:57:28.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3420 12 1832 1820 0.0332558 0 - 0.957344 2024-04-16T15:57:29.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3421 12 1832 1820 0.0332461 0 - 0.957344 2024-04-16T15:57:30.945 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3422 12 1832 1820 0.0332364 0 - 0.957344 2024-04-16T15:57:31.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3423 12 1832 1820 0.0332267 0 - 0.957344 2024-04-16T15:57:32.950 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3424 12 1832 1820 0.033217 0 - 0.957344 2024-04-16T15:57:33.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3425 12 1832 1820 0.0332073 0 - 0.957344 2024-04-16T15:57:34.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3426 12 1832 1820 0.0331976 0 - 0.957344 2024-04-16T15:57:35.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3427 12 1832 1820 0.0331879 0 - 0.957344 2024-04-16T15:57:36.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3428 12 1832 1820 0.0331782 0 - 0.957344 2024-04-16T15:57:37.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3429 12 1832 1820 0.0331685 0 - 0.957344 2024-04-16T15:57:38.946 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3430 12 1832 1820 0.0331588 0 - 0.957344 2024-04-16T15:57:39.947 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3431 12 1832 1820 0.0331492 0 - 0.957344 2024-04-16T15:57:40.947 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3432 12 1832 1820 0.0331395 0 - 0.957344 2024-04-16T15:57:41.947 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3433 12 1832 1820 0.0331299 0 - 0.957344 2024-04-16T15:57:42.947 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3434 12 1832 1820 0.0331202 0 - 0.957344 2024-04-16T15:57:43.947 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3435 12 1832 1820 0.0331106 0 - 0.957344 2024-04-16T15:57:44.947 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3436 12 1832 1820 0.0331009 0 - 0.957344 2024-04-16T15:57:45.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3437 12 1832 1820 0.0330913 0 - 0.957344 2024-04-16T15:57:46.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3438 12 1832 1820 0.0330817 0 - 0.957344 2024-04-16T15:57:47.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3439 12 1832 1820 0.0330721 0 - 0.957344 2024-04-16T15:57:48.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout:2024-04-16T15:57:48.950584+0000 min lat: 0.0141565 max lat: 302.481 avg lat: 0.957344 2024-04-16T15:57:48.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T15:57:48.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3440 12 1832 1820 0.0330625 0 - 0.957344 2024-04-16T15:57:49.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3441 12 1832 1820 0.0330528 0 - 0.957344 2024-04-16T15:57:50.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3442 12 1832 1820 0.0330432 0 - 0.957344 2024-04-16T15:57:51.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3443 12 1832 1820 0.0330336 0 - 0.957344 2024-04-16T15:57:52.948 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3444 12 1832 1820 0.0330241 0 - 0.957344 2024-04-16T15:57:53.949 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3445 12 1832 1820 0.0330145 0 - 0.957344 2024-04-16T15:57:54.949 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3446 12 1832 1820 0.0330049 0 - 0.957344 2024-04-16T15:57:55.077 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None, False)> 2024-04-16T15:57:55.078 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T15:57:55.078 ERROR:tasks.ceph.osd.3:tried to stop a non-running daemon 2024-04-16T15:57:55.081 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T15:57:55.081 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T15:57:55.081 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:57:55.083 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T15:57:55.083 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T15:57:55.083 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T15:57:55.084 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T15:57:55.084 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T15:57:55.084 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T15:57:55.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-16T15:57:55.324 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T15:57:55.324 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T15:57:55.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T15:57:55.425 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T15:57:55.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T15:57:55.483 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T15:57:55.605 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T15:57:55.606 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-16T15:57:55.671 INFO:tasks.ceph:Archiving mon data... 2024-04-16T15:57:55.673 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/data/mon.a.tgz 2024-04-16T15:57:55.674 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-04-16T15:57:55.691 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.GhTQ1VFRUw 2024-04-16T15:57:55.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GhTQ1VFRUw 2024-04-16T15:57:55.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.GhTQ1VFRUw 2024-04-16T15:57:55.903 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.GhTQ1VFRUw is 240KB 2024-04-16T15:57:55.949 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3447 12 1832 1820 0.0329953 0 - 0.957344 2024-04-16T15:57:55.997 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.GhTQ1VFRUw 2024-04-16T15:57:56.012 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T15:57:56.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T15:57:56.054 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T15:57:56.140 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T15:57:56.159 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T15:57:56.159 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T15:57:56.161 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/remote/smithi097/crash 2024-04-16T15:57:56.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T15:57:56.209 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/remote/smithi186/crash 2024-04-16T15:57:56.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T15:57:56.254 INFO:tasks.ceph:Compressing logs... 2024-04-16T15:57:56.254 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T15:57:56.258 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T15:57:56.321 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30365.log 2024-04-16T15:57:56.321 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30217.log 2024-04-16T15:57:56.321 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30365.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T15:57:56.321 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30365.log.gz 2024-04-16T15:57:56.321 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33024.log 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32124.log 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33361.log 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33024.log.gz 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33054.log 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32124.log.gz 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32513.log 2024-04-16T15:57:56.322 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33361.log.gz 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31987.log 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.33054.log.gz 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30551.log 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32513.log.gz 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.31987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29809.log 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31891.log 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30551.log.gz 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32901.log 2024-04-16T15:57:56.323 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29809.log.gz 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31891.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31891.log.gz 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30719.log 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30548.log 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32901.log.gz 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32994.log 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30719.log.gz 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30886.log 2024-04-16T15:57:56.324 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30548.log.gz 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30407.log 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32994.log.gz 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33176.log 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30886.log.gz 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30550.log 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30407.log.gz 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31373.log 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33176.log.gz 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32693.log 2024-04-16T15:57:56.325 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30550.log.gz 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31243.log 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31373.log.gz 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32931.log 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32693.log.gz 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32361.log 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31243.log.gz 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31813.log 2024-04-16T15:57:56.326 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.32931.log.gz 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30716.log 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32361.log.gz 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30549.log 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31813.log.gz 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32734.log 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30716.log.gz 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32084.log 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30549.log.gz 2024-04-16T15:57:56.327 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32554.log 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32734.log.gz 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32963.log 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32084.log.gz 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31015.log 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32554.log.gz 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32963.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.32963.log.gz 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T15:57:56.328 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33146.log 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31015.log.gz 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31163.log 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33146.log: 86.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr:gzip 8.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33146.log.gz -- 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr: /var/log/ceph/ceph-client.admin.32817.log 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29994.log 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31163.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31163.log.gz 2024-04-16T15:57:56.329 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31056.log 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32817.log.gz 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31499.log 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29994.log.gz 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30315.log 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31056.log.gz 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31667.log 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31499.log.gz 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33116.log 2024-04-16T15:57:56.330 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30315.log.gz 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33423.log 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31667.log.gz 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32413.log 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33116.log.gz 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33423.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33423.log.gz 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32164.log 2024-04-16T15:57:56.331 INFO:teuthology.orchestra.run.smithi097.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.32413.log.gz 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30827.log 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32164.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32164.log.gz 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31709.log 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33606.log 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30827.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31709.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31709.log.gz 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33513.log 2024-04-16T15:57:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33453.log 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33606.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33606.log.gz 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33513.log: gzip 8.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33513.log.gz 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31351.log 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32859.log 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33453.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33453.log.gz 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31541.log 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31351.log.gz 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29730.log 2024-04-16T15:57:56.333 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32859.log.gz 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32650.log 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31541.log.gz 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32774.log 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29730.log.gz 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33544.log 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32650.log.gz 2024-04-16T15:57:56.334 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32774.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32774.log.gz 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33299.log 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33269.log 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33544.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.33544.log.gz 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30449.log 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.33299.log.gz 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33269.log: 8.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.33269.log.gz 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31964.log 2024-04-16T15:57:56.335 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30995.log 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30449.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31964.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31964.log.gz 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30887.log 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28661.log 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30995.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30995.log.gz 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30887.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.30887.log.gz 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31242.log 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30718.log 2024-04-16T15:57:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.tmp-client.admin.28661.log: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28661.log.gz 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33236.log 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31242.log.gz 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31057.log 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30718.log.gz 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32252.log 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33236.log.gz 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31411.log 2024-04-16T15:57:56.337 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31057.log.gz 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33329.log 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32252.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32252.log.gz 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31751.log 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31411.log.gz 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33329.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33329.log.gz 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32044.log 2024-04-16T15:57:56.338 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31751.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31751.log.gz 2024-04-16T15:57:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33084.log 2024-04-16T15:57:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30273.log 2024-04-16T15:57:56.339 INFO:teuthology.orchestra.run.smithi097.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32044.log.gz 2024-04-16T15:57:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33391.log 2024-04-16T15:57:56.341 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.33084.log.gz 2024-04-16T15:57:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30273.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30273.log.gz 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31183.log 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33391.log.gz 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30885.log 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31183.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31183.log.gz 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30717.log 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30885.log: gzip -5 49.6% --verbose -- /var/log/ceph/ceph-client.admin.31892.log 2024-04-16T15:57:56.342 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30885.log.gz 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33483.log 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30717.log.gz 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33576.log 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31892.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31892.log.gz 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33206.log 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.33483.log.gz 2024-04-16T15:57:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29966.log 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.33576.log.gz 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32299.log 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33206.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.33206.log.gz 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31583.log 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29966.log.gz 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31625.log 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32299.log.gz 2024-04-16T15:57:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31583.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31583.log.gz 2024-04-16T15:57:56.345 INFO:teuthology.orchestra.run.smithi097.stderr: /var/log/ceph/ceph-client.admin.32205.log 2024-04-16T15:57:56.345 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31625.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31625.log.gz 2024-04-16T15:57:56.345 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31331.log 2024-04-16T15:57:56.345 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32205.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32205.log.gz 2024-04-16T15:57:56.345 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31331.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31331.log.gz 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29949.log 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29989.log 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29949.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29989.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29949.log.gz 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29909.log 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29989.log.gz 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T15:57:56.348 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29869.log 2024-04-16T15:57:56.349 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29909.log.gz 2024-04-16T15:57:56.349 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T15:57:56.349 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29869.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29869.log.gz 2024-04-16T15:57:56.355 INFO:teuthology.orchestra.run.smithi097.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.0.31987.log.gz 2024-04-16T15:57:56.372 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T15:57:56.409 INFO:teuthology.orchestra.run.smithi097.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T15:57:56.689 INFO:teuthology.orchestra.run.smithi186.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-16T15:57:56.769 INFO:teuthology.orchestra.run.smithi097.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T15:57:56.868 INFO:teuthology.orchestra.run.smithi097.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T15:57:56.870 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-16T15:57:56.870 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m0.600s 2024-04-16T15:57:56.871 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m1.221s 2024-04-16T15:57:56.871 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.105s 2024-04-16T15:57:56.949 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3448 12 1832 1820 0.0329857 0 - 0.957344 2024-04-16T15:57:56.975 INFO:teuthology.orchestra.run.smithi186.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T15:57:56.977 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-16T15:57:56.977 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m0.659s 2024-04-16T15:57:56.978 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m1.020s 2024-04-16T15:57:56.978 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.046s 2024-04-16T15:57:56.978 INFO:tasks.ceph:Archiving logs... 2024-04-16T15:57:56.978 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/remote/smithi097/log 2024-04-16T15:57:56.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T15:57:57.296 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/remote/smithi186/log 2024-04-16T15:57:57.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T15:57:57.542 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T15:57:57.566 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T15:57:57.575 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_Matan-B_ceph_372751789be1725ed06fd077d029295c8cab4c35/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T15:57:57.592 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T15:57:57.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T15:57:57.595 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T15:57:57.673 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T15:57:57.683 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T15:57:57.683 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T15:57:57.686 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T15:57:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-04-16T15:57:57.700 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-16T15:57:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T15:57:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-04-16T15:57:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +2455us[+2455us] +/- 83ms 2024-04-16T15:57:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 7 377 72 -1710us[-1696us] +/- 63ms 2024-04-16T15:57:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +62us[ +62us] +/- 76ms 2024-04-16T15:57:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T15:57:57.705 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T15:57:57.705 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-16T15:57:57.705 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 1 +2443us[+2443us] +/- 83ms 2024-04-16T15:57:57.705 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 7 377 1 -1732us[-1736us] +/- 64ms 2024-04-16T15:57:57.706 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +75us[ +72us] +/- 76ms 2024-04-16T15:57:57.706 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T15:57:57.706 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T15:57:57.716 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T15:57:57.717 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T15:57:57.857 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T15:57:57.925 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T15:57:57.949 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3449 12 1832 1820 0.0329762 0 - 0.957344 2024-04-16T15:57:57.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T15:57:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:57:58.025 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T15:57:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:57:58.056 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T15:57:58.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T15:57:58.076 INFO:teuthology.task.internal:Duration was 4092.927228 seconds 2024-04-16T15:57:58.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T15:57:58.085 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T15:57:58.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T15:57:58.088 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T15:57:58.131 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T15:57:58.142 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T15:57:58.394 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T15:57:58.394 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-04-16T15:57:58.395 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T15:57:58.415 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-16T15:57:58.416 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T15:57:58.442 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T15:57:58.443 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T15:57:58.458 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T15:57:58.513 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T15:57:58.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T15:57:58.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T15:57:58.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T15:57:58.671 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T15:57:58.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T15:57:58.700 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T15:57:58.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T15:57:58.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T15:57:58.775 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T15:57:58.799 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-04-16T15:57:58.802 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-16T15:57:58.839 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T15:57:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:57:58.878 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T15:57:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T15:57:58.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T15:57:58.904 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T15:57:58.905 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/remote/smithi097 2024-04-16T15:57:58.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T15:57:58.956 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391/remote/smithi186 2024-04-16T15:57:58.956 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T15:57:58.956 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3450 12 1832 1820 0.0329666 0 - 0.957344 2024-04-16T15:57:58.998 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T15:57:58.999 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T15:57:59.002 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T15:57:59.045 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T15:57:59.059 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T15:57:59.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T15:57:59.071 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T15:57:59.072 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T15:57:59.074 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T15:57:59.087 INFO:teuthology.orchestra.run.smithi097.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 15:57 /home/ubuntu/cephtest 2024-04-16T15:57:59.101 INFO:teuthology.orchestra.run.smithi186.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 15:57 /home/ubuntu/cephtest 2024-04-16T15:57:59.102 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T15:57:59.115 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T15:57:59.126 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T15:57:59.127 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T15:57:59.147 DEBUG:teuthology.nuke:shortname: smithi097 2024-04-16T15:57:59.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T15:57:59.167 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 14:40:01.376396', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNXmzu0aqrX/brCJad7hOOc5xrCmRP7WJ6BEU7E69d9quoupN6W7mtUvqmVhcN3YHewmLUVRlhtjINfAKQI77AI='} 2024-04-16T15:57:59.171 DEBUG:teuthology.nuke:shortname: smithi186 2024-04-16T15:57:59.171 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T15:57:59.192 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 14:40:01.377640', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjG692fe51fvXOSbVE4fA2zMap3ZdmEsjONTdJZdF9fG3q77p0KwYF+U68bWtBouhDOJwkcwHw+f4VGRs5dCew='} 2024-04-16T15:57:59.225 INFO:teuthology.orchestra.console.smithi097:Power off 2024-04-16T15:57:59.226 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T15:57:59.251 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-04-16T15:57:59.251 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:57:59.269 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-16T15:57:59.270 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T15:57:59.280 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-16T15:57:59.294 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-16T15:57:59.294 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:57:59.318 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-16T15:57:59.950 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3451 12 1832 1820 0.0329571 0 - 0.957344 2024-04-16T15:58:00.950 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3452 12 1832 1820 0.0329475 0 - 0.957344 2024-04-16T15:58:01.950 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3453 12 1832 1820 0.032938 0 - 0.957344 2024-04-16T15:58:02.950 INFO:tasks.radosbench.radosbench.0.smithi097.stdout: 3454 12 1832 1820 0.0329284 0 - 0.957344 2024-04-16T15:58:03.281 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:58:03.319 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:58:03.392 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-16T15:58:03.443 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-16T15:58:07.395 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:58:07.444 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:58:07.507 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-04-16T15:58:07.555 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-16T15:58:07.609 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-04-16T15:58:11.559 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T15:58:11.670 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-16T15:58:11.771 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-16T15:58:11.876 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} duration: 4092.9272282123566 failure_reason: reached maximum tries (501) after waiting for 3000 seconds flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2c1d76f6d04b413e8bcd12ad16fd1b92 status: fail success: false 2024-04-16T15:58:11.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T15:58:11.964 INFO:teuthology.run:FAIL