2024-04-16T07:32:24.956 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T07:32:24.956 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001 --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} --owner scheduled_matan@teuthology --verbose --name matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi -- /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001/orig.config.yaml 2024-04-16T07:32:25.002 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:32:25.094 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001 branch: wip-matanb-crimson-replicated-april-15 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} email: null first_in_suite: false job_id: '7658001' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-replicated-april-15 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: a7639cb6ce633dabc4d37769d62caacb54582984 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6444 sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: a7639cb6ce633dabc4d37769d62caacb54582984 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAh+NAX5OpG82UzrRzEWwlhr6vFazL5MfrxVcf8YGydbE0YmOIaI9yCh15FwPb7TAQyFusgjtw7Xn6frgxzzbIg= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_07:26:08 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T07:32:25.094 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa; will attempt to use it 2024-04-16T07:32:25.095 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks 2024-04-16T07:32:25.095 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T07:32:25.097 INFO:teuthology.task.internal:Checking packages... 2024-04-16T07:32:25.120 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '30b0aa0009a925d6a46536e55c5656edcd51ecdb' 2024-04-16T07:32:25.120 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:32:25.120 INFO:teuthology.packaging:ref: None 2024-04-16T07:32:25.121 INFO:teuthology.packaging:tag: None 2024-04-16T07:32:25.121 INFO:teuthology.packaging:branch: wip-matanb-crimson-replicated-april-15 2024-04-16T07:32:25.121 INFO:teuthology.packaging:sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:32:25.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-replicated-april-15 2024-04-16T07:32:25.336 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3109.g30b0aa00 2024-04-16T07:32:25.338 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T07:32:25.354 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T07:32:25.354 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T07:32:25.382 INFO:teuthology.task.internal:Saving configuration 2024-04-16T07:32:25.399 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T07:32:25.410 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:32:25.442 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:27:22.748715', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAh+NAX5OpG82UzrRzEWwlhr6vFazL5MfrxVcf8YGydbE0YmOIaI9yCh15FwPb7TAQyFusgjtw7Xn6frgxzzbIg='} 2024-04-16T07:32:25.442 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T07:32:25.453 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-16T07:32:25.453 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T07:32:25.497 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f048b1f2e50>, signals=[15]) 2024-04-16T07:32:25.498 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T07:32:25.508 INFO:teuthology.task.internal:Opening connections... 2024-04-16T07:32:25.509 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:32:25.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:32:25.620 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T07:32:25.627 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-04-16T07:32:25.693 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-04-16T07:32:25.693 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:32:25.751 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:32:25.752 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:32:25.752 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:32:25.752 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:32:25.752 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:32:25.752 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-04-16T07:32:25.778 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T07:32:25.789 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T07:32:25.801 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T07:32:25.801 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:32:25.819 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T07:32:25.826 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T07:32:25.826 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:32:25.879 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T07:32:25.913 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T07:32:25.933 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T07:32:25.933 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T07:32:25.933 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:32:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:32:25.951 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-04-16T07:32:26.009 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:32:26.009 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-04-16T07:32:26.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-04-16T07:32:29.139 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-16T07:32:30.615 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-04-16T07:32:31.892 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-04-16T07:32:54.226 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 914 kB/s | 19 MB 00:21 2024-04-16T07:32:58.444 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-16T07:33:00.079 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-16T07:33:02.304 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-16T07:33:07.710 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-16T07:33:10.193 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:33:10.193 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:33:10.298 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:33:10.314 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:33:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:33:10.315 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:33:10.315 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:33:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:33:10.315 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:33:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:33:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:33:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:33:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:33:10.316 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:33:10.316 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:33:10.316 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-04-16T07:33:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:33:10.317 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 77 M 2024-04-16T07:33:10.317 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 123 M 2024-04-16T07:33:10.317 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:33:10.496 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.5 MB/s | 277 kB 00:00 2024-04-16T07:33:11.382 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-04-16T07:33:12.282 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-16T07:33:12.732 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 12 MB/s | 28 MB 00:02 2024-04-16T07:33:12.736 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:33:12.736 INFO:teuthology.orchestra.run.smithi029.stdout:Total 32 MB/s | 77 MB 00:02 2024-04-16T07:33:12.736 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:33:13.198 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:33:13.199 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:33:14.050 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:33:14.050 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:33:16.477 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:33:17.180 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:33:17.305 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:33:20.163 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:33:20.348 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:33:25.428 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:33:25.494 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:33:30.591 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:03.553 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:34:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:34:03.598 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:03.965 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:03.965 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:34:03.965 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:34:03.965 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:34:04.394 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:34:04.395 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:34:04.854 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:34:04.995 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:34:04.996 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:34:04.996 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:34:04.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:34:04.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:34:04.996 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T07:34:04.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:34:04.997 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:34:05.080 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:34:05.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:34:05.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:34:05.126 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:34:05.127 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:34:05.128 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-04-16T07:34:05.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:34:05.128 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:34:05.129 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:34:05.129 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:34:05.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:34:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-04-16T07:34:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:34:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-04-16T07:34:07.675 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-04-16T07:34:07.691 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.EVtHVKjME0 2024-04-16T07:34:07.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EVtHVKjME0 2024-04-16T07:34:07.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.EVtHVKjME0 2024-04-16T07:34:07.914 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.EVtHVKjME0 is 6KB 2024-04-16T07:34:07.966 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.EVtHVKjME0 2024-04-16T07:34:07.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:34:08.048 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:34:08.049 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:34:08.049 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:34:08.049 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:34:08.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:34:08.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-04-16T07:34:08.304 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-16T07:34:38.230 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T07:34:38.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-16T07:34:38.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:34:56.799 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-04-16T07:35:05.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-16T07:35:05.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:36:05.867 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T07:36:17.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-16T07:36:17.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:36:18.304 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-16T07:36:18.766 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-16T07:36:18.766 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:36:18.766 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-04-16T07:36:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:36:18.785 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:36:18.785 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:36:18.785 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:36:19.786 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T07:36:19.786 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:36:19.875 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-04-16T07:36:19.914 DEBUG:teuthology.parallel:result is None 2024-04-16T07:36:19.915 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T07:36:19.937 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T07:36:19.937 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:36:19.956 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T07:36:19.972 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T07:36:19.989 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T07:36:19.989 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:36:20.032 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T07:36:20.044 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T07:36:20.044 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:36:20.104 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:36:20.120 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:36:20.122 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T07:36:20.132 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T07:36:20.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:36:20.164 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T07:36:20.178 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T07:36:20.179 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:36:20.225 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:36:20.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:36:20.366 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:36:20.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:36:20.495 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:36:20.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:36:20.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-04-16T07:36:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:36:21.023 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T07:36:21.035 INFO:teuthology.task.internal:Starting timer... 2024-04-16T07:36:21.035 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T07:36:21.066 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T07:36:21.079 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T07:36:21.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-04-16T07:36:21.153 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-04-16T07:36:21.156 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T07:36:21.156 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-04-16T07:36:21.197 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-04-16T07:36:21.198 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive'} 2024-04-16T07:36:21.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:36:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:36:21.267 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T07:36:21.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:36:21.350 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T07:36:21.367 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T07:36:21.404 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T07:36:21.405 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T07:41:00.647 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com')] 2024-04-16T07:41:00.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-16T07:41:00.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:41:00.756 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-16T07:41:00.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-16T07:41:00.849 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T07:41:00.859 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T07:41:00.860 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:41:00.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:41:00.924 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:41:00.942 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:41:00.974 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-04-16T07:41:00.988 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-04-16T07:41:01.006 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:41:01.081 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:41:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:41:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-16T07:41:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:01.085 INFO:teuthology.run_tasks:Running task install... 2024-04-16T07:41:01.094 DEBUG:teuthology.task.install:project ceph 2024-04-16T07:41:01.094 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:41:01.095 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'} 2024-04-16T07:41:01.095 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T07:41:01.100 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T07:41:01.100 INFO:teuthology.task.install:extra packages: [] 2024-04-16T07:41:01.101 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:41:01.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:41:01.356 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:41:01.357 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:41:01.781 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:41:01.781 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:41:01.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:41:01.814 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:41:01.814 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:41:01.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:41:01.971 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:41:02.041 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-04-16T07:41:02.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-04-16T07:41:02.292 INFO:teuthology.orchestra.run.smithi029.stdout:74 files removed 2024-04-16T07:41:02.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-04-16T07:41:03.770 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 77 kB/s | 87 kB 00:01 2024-04-16T07:41:05.697 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 6.2 kB/s | 12 kB 00:01 2024-04-16T07:41:06.508 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 2.7 kB/s | 2.0 kB 00:00 2024-04-16T07:41:07.010 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-16T07:41:08.511 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-04-16T07:41:09.747 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-04-16T07:41:12.252 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-16T07:41:16.610 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-16T07:41:18.270 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-16T07:41:19.966 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-16T07:41:25.268 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-16T07:41:28.193 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:41:28.212 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:41:28.212 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:41:28.212 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:41:28.212 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:41:28.213 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:41:28.214 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:41:28.215 INFO:teuthology.orchestra.run.smithi029.stdout:Install 30 Packages 2024-04-16T07:41:28.216 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-04-16T07:41:28.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:41:28.216 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 59 M 2024-04-16T07:41:28.216 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:41:43.146 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 411 kB/s | 5.9 MB 00:14 2024-04-16T07:41:43.388 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 104 kB/s | 25 kB 00:00 2024-04-16T07:41:44.889 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 531 kB/s | 797 kB 00:01 2024-04-16T07:41:46.048 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): libradosstriper1-19.0.0-3109.g30b0aa00. 471 kB/s | 546 kB 00:01 2024-04-16T07:41:58.582 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_6 428 kB/s | 5.2 MB 00:12 2024-04-16T07:41:58.757 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 257 kB/s | 45 kB 00:00 2024-04-16T07:41:59.474 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 188 kB/s | 135 kB 00:00 2024-04-16T07:42:00.041 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 281 kB/s | 159 kB 00:00 2024-04-16T07:42:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): python3-rados-19.0.0-3109.g30b0aa00.el9 196 kB/s | 315 kB 00:01 2024-04-16T07:42:02.984 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 220 kB/s | 293 kB 00:01 2024-04-16T07:42:03.318 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 298 kB/s | 99 kB 00:00 2024-04-16T07:42:03.437 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 897 kB/s | 106 kB 00:00 2024-04-16T07:42:03.487 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-16T07:42:03.529 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-04-16T07:42:03.597 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-04-16T07:42:03.639 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-16T07:42:03.756 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 384 kB/s | 41 kB 00:00 2024-04-16T07:42:03.798 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-04-16T07:42:03.832 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-04-16T07:42:03.874 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-16T07:42:04.258 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 805 kB/s | 309 kB 00:00 2024-04-16T07:42:04.600 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-16T07:42:04.668 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-04-16T07:42:04.735 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-04-16T07:42:04.810 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-04-16T07:42:04.902 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-04-16T07:42:04.979 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.4 MB/s | 192 kB 00:00 2024-04-16T07:42:05.088 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-16T07:42:07.047 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9 268 kB/s | 10 MB 00:38 2024-04-16T07:42:12.922 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 444 kB/s | 3.4 MB 00:07 2024-04-16T07:42:17.047 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 376 kB/s | 3.7 MB 00:09 2024-04-16T07:42:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): ceph-common-19.0.0-3109.g30b0aa00.el9. 288 kB/s | 19 MB 01:09 2024-04-16T07:42:37.542 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:37.542 INFO:teuthology.orchestra.run.smithi029.stdout:Total 874 kB/s | 59 MB 01:09 2024-04-16T07:42:37.542 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:42:37.582 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:42:37.582 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:42:38.091 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:42:38.092 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:42:39.564 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:42:40.093 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:42:40.235 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:42:40.311 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:42:40.403 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:42:40.429 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:42:40.484 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:42:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:42:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:42:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:42:40.684 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:42:40.700 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:42:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:42:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:42:41.265 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:42:41.293 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:42:41.361 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:42:41.406 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:42:41.470 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:42:41.503 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:42:41.592 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:42:41.626 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:42:41.714 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:42:41.759 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:42:41.871 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:42:41.912 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:42:41.953 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:42:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:42:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:42:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:42:42.102 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:42:42.693 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:42:42.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:42:42.804 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:42:42.884 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:42:42.911 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:42:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:42:45.523 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:42:46.233 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:42:46.272 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:42:46.353 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:42:46.354 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:42:46.354 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:42:46.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:42:46.391 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-04-16T07:42:58.573 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-04-16T07:42:58.574 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-04-16T07:42:58.574 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:42:59.676 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:42:59.700 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:42:59.700 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:42:59.700 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:42:59.700 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:42:59.700 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:42:59.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:42:59.702 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:42:59.796 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:42:59.796 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:43:01.487 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:43:01.487 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:43:01.488 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:43:01.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:43:01.490 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:43:01.490 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:43:01.490 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:43:01.491 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:43:01.491 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:43:01.491 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:43:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:43:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:43:02.537 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-04-16T07:43:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:43:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:43:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:43:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:43:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:43:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:43:02.540 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:43:02.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-04-16T07:43:03.526 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:38 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:43:03.692 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:43:03.693 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:43:03.693 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:43:03.693 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:43:03.693 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:43:03.693 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:43:03.693 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:43:03.694 INFO:teuthology.orchestra.run.smithi029.stdout:Install 11 Packages 2024-04-16T07:43:03.695 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:43:03.695 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 87 M 2024-04-16T07:43:03.695 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 814 M 2024-04-16T07:43:03.695 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:43:04.804 INFO:teuthology.orchestra.run.smithi029.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 703 kB/s | 217 kB 00:00 2024-04-16T07:43:04.896 INFO:teuthology.orchestra.run.smithi029.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-16T07:43:04.947 INFO:teuthology.orchestra.run.smithi029.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-04-16T07:43:04.989 INFO:teuthology.orchestra.run.smithi029.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-16T07:43:05.040 INFO:teuthology.orchestra.run.smithi029.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 90 kB 00:00 2024-04-16T07:43:05.133 INFO:teuthology.orchestra.run.smithi029.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-04-16T07:43:05.690 INFO:teuthology.orchestra.run.smithi029.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 839 kB/s | 469 kB 00:00 2024-04-16T07:43:05.841 INFO:teuthology.orchestra.run.smithi029.stdout:(8/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 129 kB/s | 173 kB 00:01 2024-04-16T07:43:05.867 INFO:teuthology.orchestra.run.smithi029.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-04-16T07:43:06.059 INFO:teuthology.orchestra.run.smithi029.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 560 kB/s | 122 kB 00:00 2024-04-16T07:45:14.069 INFO:teuthology.orchestra.run.smithi029.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 666 kB/s | 84 MB 02:09 2024-04-16T07:45:14.069 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:45:14.070 INFO:teuthology.orchestra.run.smithi029.stdout:Total 684 kB/s | 87 MB 02:10 2024-04-16T07:45:14.104 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:45:14.127 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:45:14.127 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:45:14.769 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:45:14.769 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:45:15.825 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:45:15.901 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:45:15.945 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:45:15.979 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:45:16.055 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:45:16.091 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:45:16.137 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:45:16.191 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:45:16.230 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:45:16.267 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:45:16.315 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:45:16.351 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:45:28.319 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:45:28.409 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:45:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:45:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:45:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:45:41.014 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:45:41.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-04-16T07:45:41.936 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:04:16 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:45:42.095 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:45:42.098 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:45:42.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:45:42.100 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout:Install 36 Packages 2024-04-16T07:45:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:45:42.102 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 84 M 2024-04-16T07:45:42.102 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 922 M 2024-04-16T07:45:42.102 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:45:43.090 INFO:teuthology.orchestra.run.smithi029.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 18 kB/s | 6.5 kB 00:00 2024-04-16T07:45:44.565 INFO:teuthology.orchestra.run.smithi029.stdout:(2/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 1.2 MB/s | 2.2 MB 00:01 2024-04-16T07:45:47.041 INFO:teuthology.orchestra.run.smithi029.stdout:(3/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86_ 1.3 MB/s | 5.0 MB 00:03 2024-04-16T07:45:47.099 INFO:teuthology.orchestra.run.smithi029.stdout:(4/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 365 kB/s | 1.6 MB 00:04 2024-04-16T07:45:47.225 INFO:teuthology.orchestra.run.smithi029.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-04-16T07:45:47.267 INFO:teuthology.orchestra.run.smithi029.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T07:45:47.301 INFO:teuthology.orchestra.run.smithi029.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 939 kB/s | 32 kB 00:00 2024-04-16T07:45:47.326 INFO:teuthology.orchestra.run.smithi029.stdout:(8/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 871 kB/s | 249 kB 00:00 2024-04-16T07:45:47.352 INFO:teuthology.orchestra.run.smithi029.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 931 kB/s | 47 kB 00:00 2024-04-16T07:45:47.377 INFO:teuthology.orchestra.run.smithi029.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 763 kB/s | 39 kB 00:00 2024-04-16T07:45:47.411 INFO:teuthology.orchestra.run.smithi029.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-04-16T07:45:47.479 INFO:teuthology.orchestra.run.smithi029.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-04-16T07:45:47.504 INFO:teuthology.orchestra.run.smithi029.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-04-16T07:45:47.531 INFO:teuthology.orchestra.run.smithi029.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.3 MB/s | 124 kB 00:00 2024-04-16T07:45:47.557 INFO:teuthology.orchestra.run.smithi029.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.5 MB/s | 187 kB 00:00 2024-04-16T07:45:47.641 INFO:teuthology.orchestra.run.smithi029.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-04-16T07:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 677 kB/s | 125 kB 00:00 2024-04-16T07:45:47.834 INFO:teuthology.orchestra.run.smithi029.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 785 kB/s | 151 kB 00:00 2024-04-16T07:45:47.859 INFO:teuthology.orchestra.run.smithi029.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 308 kB/s | 43 kB 00:00 2024-04-16T07:45:47.893 INFO:teuthology.orchestra.run.smithi029.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-16T07:45:47.927 INFO:teuthology.orchestra.run.smithi029.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-04-16T07:45:47.953 INFO:teuthology.orchestra.run.smithi029.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-04-16T07:45:47.978 INFO:teuthology.orchestra.run.smithi029.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 348 kB/s | 18 kB 00:00 2024-04-16T07:45:48.003 INFO:teuthology.orchestra.run.smithi029.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 458 kB/s | 23 kB 00:00 2024-04-16T07:45:48.029 INFO:teuthology.orchestra.run.smithi029.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-04-16T07:45:48.055 INFO:teuthology.orchestra.run.smithi029.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 388 kB/s | 20 kB 00:00 2024-04-16T07:45:48.080 INFO:teuthology.orchestra.run.smithi029.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 917 kB/s | 46 kB 00:00 2024-04-16T07:45:48.105 INFO:teuthology.orchestra.run.smithi029.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-16T07:45:48.133 INFO:teuthology.orchestra.run.smithi029.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.1 MB/s | 272 kB 00:00 2024-04-16T07:45:48.150 INFO:teuthology.orchestra.run.smithi029.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 376 kB/s | 16 kB 00:00 2024-04-16T07:45:48.184 INFO:teuthology.orchestra.run.smithi029.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-16T07:45:48.201 INFO:teuthology.orchestra.run.smithi029.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 706 kB/s | 36 kB 00:00 2024-04-16T07:45:48.277 INFO:teuthology.orchestra.run.smithi029.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-16T07:45:48.311 INFO:teuthology.orchestra.run.smithi029.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-04-16T07:45:48.336 INFO:teuthology.orchestra.run.smithi029.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 345 kB/s | 20 kB 00:00 2024-04-16T07:47:27.038 INFO:teuthology.orchestra.run.smithi029.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 708 kB/s | 71 MB 01:42 2024-04-16T07:47:27.042 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:47:27.042 INFO:teuthology.orchestra.run.smithi029.stdout:Total 822 kB/s | 84 MB 01:44 2024-04-16T07:47:27.062 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:47:27.106 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:47:27.107 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:47:27.919 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:47:27.919 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:47:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:47:29.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:47:29.684 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:47:29.721 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:47:29.804 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:47:29.855 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:47:29.897 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:47:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:47:30.090 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:47:30.145 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:47:30.228 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:47:30.290 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:47:30.369 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:47:30.437 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:47:30.510 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:47:30.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:47:30.786 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:47:30.829 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:47:30.879 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:47:30.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:47:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:47:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:47:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:47:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:47:31.182 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:47:31.226 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:47:31.328 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:47:31.384 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:47:31.468 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:47:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:47:31.761 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:47:31.968 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:47:32.003 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:47:32.028 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:47:32.028 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:32.028 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:47:32.028 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:47:32.028 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:47:32.028 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:47:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:47:43.491 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:47:43.491 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:43.491 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:47:43.491 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:47:43.491 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:47:43.491 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:47:44.066 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:47:44.088 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:47:44.088 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:44.088 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:47:44.089 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:47:44.089 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:47:44.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:47:44.344 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:47:44.367 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:47:44.367 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:44.367 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:47:44.367 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:47:44.367 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:47:44.367 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:47:44.444 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:47:57.910 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:47:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:47:57.912 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:47:57.913 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:47:58.809 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:47:58.810 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:47:58.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:47:58.812 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:47:59.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-04-16T07:47:59.757 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:34 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:47:59.860 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:47:59.915 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:47:59.916 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:47:59.916 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:47:59.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-04-16T07:48:00.579 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:35 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:00.735 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 762 k 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 769 k 2024-04-16T07:48:00.737 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:01.181 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-16T07:48:01.182 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:01.182 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-16T07:48:01.182 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:01.188 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:01.188 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:01.194 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:01.195 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:01.275 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:01.756 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:01.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:02.325 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:02.623 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:02.623 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:02.623 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:02.623 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:02.623 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:02.624 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:02.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:48:03.471 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:38 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:03.628 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:03.628 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:03.628 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:48:03.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:03.630 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 157 k 2024-04-16T07:48:03.630 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 640 k 2024-04-16T07:48:03.630 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:03.865 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 670 kB/s | 157 kB 00:00 2024-04-16T07:48:03.865 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:03.866 INFO:teuthology.orchestra.run.smithi029.stdout:Total 666 kB/s | 157 kB 00:00 2024-04-16T07:48:03.866 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:03.876 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:03.876 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:03.932 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:03.932 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:04.158 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:48:04.184 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:48:04.184 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:48:04.184 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:48:04.185 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:05.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:48:05.281 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:05.281 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:05.281 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:05.281 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:05.281 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:05.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-04-16T07:48:06.160 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:41 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:06.262 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:06.317 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:06.318 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:06.318 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:06.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:48:06.993 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:41 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:07.152 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:48:07.153 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout:Install 11 Packages 2024-04-16T07:48:07.154 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:07.155 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 7.6 M 2024-04-16T07:48:07.155 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 100 M 2024-04-16T07:48:07.155 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:07.843 INFO:teuthology.orchestra.run.smithi029.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3109.g30 169 kB/s | 34 kB 00:00 2024-04-16T07:48:07.960 INFO:teuthology.orchestra.run.smithi029.stdout:(2/11): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-04-16T07:48:08.129 INFO:teuthology.orchestra.run.smithi029.stdout:(3/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa0 7.3 MB/s | 3.5 MB 00:00 2024-04-16T07:48:08.153 INFO:teuthology.orchestra.run.smithi029.stdout:(4/11): ceph-prometheus-alerts-19.0.0-3109.g30b 32 kB/s | 16 kB 00:00 2024-04-16T07:48:08.287 INFO:teuthology.orchestra.run.smithi029.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-04-16T07:48:08.412 INFO:teuthology.orchestra.run.smithi029.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-16T07:48:08.638 INFO:teuthology.orchestra.run.smithi029.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 644 kB/s | 144 kB 00:00 2024-04-16T07:48:08.672 INFO:teuthology.orchestra.run.smithi029.stdout:(8/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-04-16T07:48:08.739 INFO:teuthology.orchestra.run.smithi029.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 305 kB/s | 31 kB 00:00 2024-04-16T07:48:08.774 INFO:teuthology.orchestra.run.smithi029.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-04-16T07:48:09.008 INFO:teuthology.orchestra.run.smithi029.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.8 MB/s | 2.0 MB 00:00 2024-04-16T07:48:09.014 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:09.014 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.1 MB/s | 7.6 MB 00:01 2024-04-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:09.051 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:09.051 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:09.229 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:09.229 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:48:09.829 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:48:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:48:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:48:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:48:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:48:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:48:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:48:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:48:11.443 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:48:11.491 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:48:13.790 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:48:13.791 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:48:13.791 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:48:13.791 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:48:13.791 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:48:14.136 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:14.137 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:14.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:48:15.057 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:50 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:15.217 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:15.218 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-04-16T07:48:15.218 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:48:15.219 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:15.220 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2024-04-16T07:48:15.221 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-04-16T07:48:15.221 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:15.768 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 204 kB/s | 32 kB 00:00 2024-04-16T07:48:15.819 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 347 kB/s | 17 kB 00:00 2024-04-16T07:48:15.861 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 820 kB/s | 34 kB 00:00 2024-04-16T07:48:15.953 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.8 MB/s | 3.0 MB 00:00 2024-04-16T07:48:16.322 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-04-16T07:48:16.431 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 4.8 MB 00:00 2024-04-16T07:48:16.464 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.5 MB/s | 217 kB 00:00 2024-04-16T07:48:16.740 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-31 6.6 MB/s | 7.4 MB 00:01 2024-04-16T07:48:17.182 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-04-16T07:48:17.225 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-16T07:48:17.275 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 367 kB/s | 195 kB 00:00 2024-04-16T07:48:17.517 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 734 kB/s | 245 kB 00:00 2024-04-16T07:48:17.520 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:17.520 INFO:teuthology.orchestra.run.smithi029.stdout:Total 16 MB/s | 37 MB 00:02 2024-04-16T07:48:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:17.590 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:17.590 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:17.967 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:17.968 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:18.692 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:18.783 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:48:18.823 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:48:18.938 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:48:18.988 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:48:19.173 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:48:19.212 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:48:19.300 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:48:20.008 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:48:20.136 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:48:21.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:48:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:48:22.577 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:48:26.804 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:48:26.804 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:48:26.804 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:48:26.805 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:48:27.240 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:48:27.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:48:27.242 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:48:27.242 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:48:27.242 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:27.242 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:27.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:48:28.308 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:03 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:28.466 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:28.467 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:48:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-04-16T07:48:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-04-16T07:48:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-04-16T07:48:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:28.792 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 24 kB 00:00 2024-04-16T07:48:28.817 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-04-16T07:48:28.894 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-04-16T07:48:28.920 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-04-16T07:48:28.945 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 176 kB/s | 49 kB 00:00 2024-04-16T07:48:28.979 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.4 MB/s | 211 kB 00:00 2024-04-16T07:48:29.004 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 507 kB/s | 43 kB 00:00 2024-04-16T07:48:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-04-16T07:48:29.289 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-04-16T07:48:29.506 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 404 kB/s | 203 kB 00:00 2024-04-16T07:48:29.540 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 235 kB/s | 59 kB 00:00 2024-04-16T07:48:29.682 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 510 kB/s | 90 kB 00:00 2024-04-16T07:48:29.816 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-04-16T07:48:29.818 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:29.818 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-04-16T07:48:29.819 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:29.835 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:29.835 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:29.938 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:29.939 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:30.462 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:30.647 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:48:30.706 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:48:30.770 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:48:30.815 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:48:30.865 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:48:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:48:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:48:31.139 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:48:31.903 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:48:31.965 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:48:31.999 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:48:32.049 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:48:32.086 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:48:33.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:48:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:48:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:48:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:48:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:48:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:48:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:48:33.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:33.578 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:33.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:48:34.537 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:09 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:34.698 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:34.698 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-04-16T07:48:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-04-16T07:48:34.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2024-04-16T07:48:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-04-16T07:48:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:35.378 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-16T07:48:35.512 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-04-16T07:48:35.571 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 380 kB/s | 142 kB 00:00 2024-04-16T07:48:35.838 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 191 kB/s | 58 kB 00:00 2024-04-16T07:48:35.863 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 174 kB/s | 51 kB 00:00 2024-04-16T07:48:35.972 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 927 kB/s | 548 kB 00:00 2024-04-16T07:48:35.975 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:35.976 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-04-16T07:48:35.989 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:36.000 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:36.000 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:36.094 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:36.094 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:36.396 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:36.586 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:48:36.651 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:48:37.211 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:48:37.301 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:48:37.373 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:48:37.409 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:48:38.551 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:48:38.552 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:48:38.552 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:48:38.552 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:48:38.552 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:48:38.552 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:48:38.866 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:38.867 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:39.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-04-16T07:48:39.795 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:14 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:39.955 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:39.955 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:39.955 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-04-16T07:48:39.956 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:39.957 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 950 k 2024-04-16T07:48:39.957 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 4.0 M 2024-04-16T07:48:39.957 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:40.070 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 697 kB/s | 79 kB 00:00 2024-04-16T07:48:41.680 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 506 kB/s | 871 kB 00:01 2024-04-16T07:48:41.681 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:41.682 INFO:teuthology.orchestra.run.smithi029.stdout:Total 550 kB/s | 950 kB 00:01 2024-04-16T07:48:41.682 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:41.709 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:41.709 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:41.790 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:41.790 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:42.146 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:48:42.227 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:48:42.973 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:48:42.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:48:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:48:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:43.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:48:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:43.319 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:43.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2024-04-16T07:48:44.290 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:19 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:44.450 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:44.450 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-04-16T07:48:44.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:44.452 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 342 k 2024-04-16T07:48:44.452 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.5 M 2024-04-16T07:48:44.452 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:44.585 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 517 kB/s | 69 kB 00:00 2024-04-16T07:48:44.719 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 1.0 MB/s | 273 kB 00:00 2024-04-16T07:48:44.720 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:44.720 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.2 MB/s | 342 kB 00:00 2024-04-16T07:48:44.721 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:44.730 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:44.730 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:44.772 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:44.772 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:45.039 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:45.234 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:48:45.283 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:48:45.299 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:48:45.299 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:48:45.299 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:48:45.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:46.215 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:48:46.568 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:48:46.568 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:46.568 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:46.568 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:46.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:48:46.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:46.569 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:46.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-04-16T07:48:47.552 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:22 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:47.709 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:47.710 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-16T07:48:47.710 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:47.710 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:48:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 125 k 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 445 k 2024-04-16T07:48:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:48.034 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 386 kB/s | 125 kB 00:00 2024-04-16T07:48:48.035 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:48.035 INFO:teuthology.orchestra.run.smithi029.stdout:Total 384 kB/s | 125 kB 00:00 2024-04-16T07:48:48.035 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:48.041 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:48.041 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:48.100 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:48.100 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:48.361 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:48.848 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:49.151 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:49.151 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:49.151 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:49.151 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:49.151 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:49.151 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:49.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-04-16T07:48:50.056 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:25 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:50.160 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:50.216 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:50.216 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:50.217 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:50.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-04-16T07:48:50.897 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:25 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:51.057 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:51.058 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:48:51.058 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:51.058 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 32 k 2024-04-16T07:48:51.058 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 142 k 2024-04-16T07:48:51.058 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:51.346 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 110 kB/s | 32 kB 00:00 2024-04-16T07:48:51.346 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:51.346 INFO:teuthology.orchestra.run.smithi029.stdout:Total 110 kB/s | 32 kB 00:00 2024-04-16T07:48:51.347 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:51.350 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:51.350 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:51.380 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:51.381 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:51.475 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:51.614 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:52.289 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:52.290 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:52.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-04-16T07:48:53.126 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:28 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:53.230 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:53.285 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:53.286 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:53.286 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:53.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-04-16T07:48:53.972 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:28 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:54.076 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:54.131 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:54.132 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:54.132 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:54.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-04-16T07:48:54.807 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:29 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:54.910 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:54.966 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:54.967 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:54.967 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:55.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-04-16T07:48:55.656 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:30 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:55.758 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:55.813 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:55.814 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:55.815 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:55.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-04-16T07:48:56.496 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:31 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:56.600 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:56.657 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:56.658 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:56.658 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:56.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-04-16T07:48:57.342 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:32 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:57.444 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:57.500 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:57.501 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-16T07:48:57.501 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:48:57.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-04-16T07:48:58.188 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:33 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:48:58.346 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 90 k 2024-04-16T07:48:58.347 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 313 k 2024-04-16T07:48:58.348 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:48:58.679 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 271 kB/s | 90 kB 00:00 2024-04-16T07:48:58.679 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:58.680 INFO:teuthology.orchestra.run.smithi029.stdout:Total 269 kB/s | 90 kB 00:00 2024-04-16T07:48:58.680 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:48:58.689 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:48:58.689 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:48:58.747 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:48:58.747 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:48:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:48:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:59.432 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:59.731 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:59.732 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:59.732 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:48:59.732 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:59.732 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:48:59.732 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:49:00.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-04-16T07:49:00.620 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:35 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:49:00.779 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:49:00.780 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:00.780 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-16T07:49:00.780 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.4 M 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 29 M 2024-04-16T07:49:00.781 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:49:01.371 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 5.8 MB/s | 3.4 MB 00:00 2024-04-16T07:49:01.372 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:49:01.372 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.8 MB/s | 3.4 MB 00:00 2024-04-16T07:49:01.372 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:49:01.384 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:49:01.384 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:49:01.463 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:49:01.463 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:49:02.048 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:49:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:02.140 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:49:02.140 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:49:02.140 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:49:02.140 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:49:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:03.759 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:49:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:49:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:03.759 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:49:04.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-04-16T07:49:04.655 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:39 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:49:04.817 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:49:04.817 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-16T07:49:04.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:04.819 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 175 k 2024-04-16T07:49:04.819 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 673 k 2024-04-16T07:49:04.819 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:49:05.200 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 460 kB/s | 175 kB 00:00 2024-04-16T07:49:05.200 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:49:05.201 INFO:teuthology.orchestra.run.smithi029.stdout:Total 459 kB/s | 175 kB 00:00 2024-04-16T07:49:05.201 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:49:05.212 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:49:05.212 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:49:05.271 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:49:05.271 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:49:05.426 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:49:05.538 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:06.081 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:06.379 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:06.379 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:06.379 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:49:06.379 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:49:06.379 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:06.379 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:49:06.705 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:06.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:49:06.849 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:49:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:49:06.871 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:49:06.871 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:49:06.873 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T07:49:06.873 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:06.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:49:06.940 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T07:49:06.941 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:06.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:49:07.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:49:07.076 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T07:49:07.077 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:07.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:49:07.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:49:07.212 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T07:49:07.212 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:07.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:49:07.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:49:07.348 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T07:49:07.424 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T07:49:07.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-04-16T07:49:07.453 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T07:49:07.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:49:07.549 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T07:49:07.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:49:07.584 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T07:49:07.584 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T07:49:07.585 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001', 'branch': 'wip-matanb-crimson-replicated-april-15', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read}', 'email': None, 'first_in_suite': False, 'job_id': '7658001', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'a7639cb6ce633dabc4d37769d62caacb54582984'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6444, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'a7639cb6ce633dabc4d37769d62caacb54582984', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAh+NAX5OpG82UzrRzEWwlhr6vFazL5MfrxVcf8YGydbE0YmOIaI9yCh15FwPb7TAQyFusgjtw7Xn6frgxzzbIg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_07:26:08', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T07:49:07.585 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:49:07.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:49:07.701 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:07.701 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:49:07.756 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:49:07.756 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-16 07:49:02.388906821 +0000 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-16 07:40:36.186470408 +0000 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-16 07:40:36.186470408 +0000 2024-04-16T07:49:07.811 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-16 07:40:36.186470408 +0000 2024-04-16T07:49:07.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:49:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-16T07:49:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-16T07:49:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.0472e-05 s, 5.7 MB/s 2024-04-16T07:49:07.878 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:49:07.935 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-16 07:49:02.389906804 +0000 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-16 07:40:36.526464662 +0000 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-16 07:40:36.526464662 +0000 2024-04-16T07:49:07.990 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-16 07:40:36.526464662 +0000 2024-04-16T07:49:07.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:49:08.056 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-16T07:49:08.056 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-16T07:49:08.056 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.5279e-05 s, 5.4 MB/s 2024-04-16T07:49:08.057 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:49:08.113 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-04-16T07:49:08.168 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-16 07:49:02.389906804 +0000 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-16 07:40:36.852459154 +0000 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-16 07:40:36.852459154 +0000 2024-04-16T07:49:08.169 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-16 07:40:36.852459154 +0000 2024-04-16T07:49:08.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:49:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-16T07:49:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-16T07:49:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 7.4562e-05 s, 6.9 MB/s 2024-04-16T07:49:08.236 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:49:08.293 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-04-16T07:49:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:49:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:08.348 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-16T07:49:08.348 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:08.348 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:08.349 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-16 07:49:02.390906787 +0000 2024-04-16T07:49:08.349 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-16 07:40:37.186453509 +0000 2024-04-16T07:49:08.349 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-16 07:40:37.186453509 +0000 2024-04-16T07:49:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-16 07:40:37.186453509 +0000 2024-04-16T07:49:08.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:49:08.414 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-16T07:49:08.414 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-16T07:49:08.414 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.5043e-05 s, 5.4 MB/s 2024-04-16T07:49:08.415 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:49:08.471 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:08.471 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T07:49:08.471 INFO:tasks.ceph:Generating config... 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-16T07:49:08.473 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] log to file = False 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-16T07:49:08.474 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T07:49:08.474 INFO:tasks.ceph:[global] auth client required = none 2024-04-16T07:49:08.474 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-16T07:49:08.474 INFO:tasks.ceph:[global] auth service required = none 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] auth supported = none 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-16T07:49:08.475 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-16T07:49:08.476 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-16T07:49:08.477 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:49:08.478 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T07:49:08.479 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T07:49:08.479 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T07:49:08.479 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T07:49:08.479 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T07:49:08.479 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T07:49:08.479 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T07:49:08.479 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T07:49:08.479 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-16T07:49:08.479 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-16T07:49:08.479 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-16T07:49:08.479 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T07:49:08.479 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-16T07:49:08.479 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-16T07:49:08.479 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-16T07:49:08.479 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-16T07:49:08.480 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T07:49:08.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T07:49:08.570 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T07:49:08.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T07:49:08.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:49:08.654 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29')] 2024-04-16T07:49:08.654 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-16T07:49:08.655 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:08.655 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:49:08.709 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:08.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:08.800+0000 7ff225348f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:08.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:08.800+0000 7ff225348f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:08.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:08.800+0000 7ff225348f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:08.801 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:08.801 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 37726edf-a428-419e-bc09-16ace721206f 2024-04-16T07:49:08.802 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-04-16T07:49:08.802 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2024-04-16T07:49:08.802 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 37726edf-a428-419e-bc09-16ace721206f 2024-04-16T07:49:08.802 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2024-04-16T07:49:08.802253+0000 2024-04-16T07:49:08.802 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-16T07:49:08.802253+0000 2024-04-16T07:49:08.803 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 17 (quincy) 2024-04-16T07:49:08.803 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2024-04-16T07:49:08.803 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-04-16T07:49:08.803 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T07:49:08.804 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:49:08.819 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 37726edf-a428-419e-bc09-16ace721206f... 2024-04-16T07:49:08.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = none 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2024-04-16T07:49:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = 172.21.15.29 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: auth client required = none 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: auth cluster required = none 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: auth service required = none 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: debug asok = 0/0 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 0/0 2024-04-16T07:49:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: debug bdev = 0/0 2024-04-16T07:49:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 0/0 2024-04-16T07:49:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 0/0 2024-04-16T07:49:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: debug buffer = 0/0 2024-04-16T07:49:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: debug civetweb = 0/0 2024-04-16T07:49:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 0/0 2024-04-16T07:49:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: debug compressor = 0/0 2024-04-16T07:49:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: debug context = 0/0 2024-04-16T07:49:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: debug crush = 0/0 2024-04-16T07:49:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: debug crypto = 0/0 2024-04-16T07:49:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: debug dpdk = 0/0 2024-04-16T07:49:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: debug eventtrace = 0/0 2024-04-16T07:49:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: debug filer = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug filestore = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug finisher = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug fuse = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug heartbeatmap = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug javaclient = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug journal = 0/0 2024-04-16T07:49:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: debug journaler = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug kstore = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug leveldb = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug lockdep = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds balancer = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds locker = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds log = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds log expire = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds migrator = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug memdb = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgrc = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 0/0 2024-04-16T07:49:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: debug objclass = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug objectcacher = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug objecter = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug optracker = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug perfcounter = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rados = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd mirror = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd replay = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug refs = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rgw = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rgw lifecycle = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rgw notification = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rgw sync = 0/0 2024-04-16T07:49:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 0/0 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: debug striper = 0/0 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: debug throttle = 0/0 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: debug timer = 0/0 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: debug tp = 0/0 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: ms client mode = crc 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: ms cluster mode = crc 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: ms mon client mode = crc 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: ms mon cluster mode = crc 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: ms mon service mode = crc 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: ms service mode = crc 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 37726edf-a428-419e-bc09-16ace721206f 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2024-04-16T07:49:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:49:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression algorithm = snappy 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression mode = aggressive 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore rocksdb cf = False 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:49:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: crimson osd obc lru size = 10 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 20 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 20 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug filestore = 0/0 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug journal = 0/0 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 0/0 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 0/0 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: err to stderr = True 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: log flush on exit = True 2024-04-16T07:49:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: log to file = False 2024-04-16T07:49:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: log to stderr = True 2024-04-16T07:49:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:49:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2024-04-16T07:49:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:49:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: osd shutdown pgref assert = True 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2024-04-16T07:49:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-04-16T07:49:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-04-16T07:49:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:49:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:49:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2024-04-16T07:49:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.928 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2024-04-16T07:49:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 0/0 2024-04-16T07:49:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 0/0 2024-04-16T07:49:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 0/0 2024-04-16T07:49:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-04-16T07:49:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:49:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default crimson = True 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2024-04-16T07:49:08.931 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2024-04-16T07:49:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:49:08.933 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T07:49:08.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T07:49:09.041 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T07:49:09.041 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:09.041 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T07:49:09.058 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:09.058 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T07:49:09.113 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:49:09.113 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:09.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:49:09.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:49:09.194 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:09.194 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:09.249 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T07:49:09.249 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T07:49:09.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:49:09.354 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:49:09.357 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T07:49:09.357 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T07:49:09.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T07:49:09.408 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T07:49:09.425 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T07:49:09.425 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T07:49:09.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:09.452 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:09.452 INFO:tasks.ceph:role: osd.0 2024-04-16T07:49:09.452 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:49:09.453 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:49:09.520 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:49:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:49:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:49:09.521 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:49:09.552 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-16T07:49:09.554 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-16T07:49:09.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:09.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:09.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:09.769 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:09.769 INFO:tasks.ceph:role: osd.1 2024-04-16T07:49:09.769 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:49:09.769 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:49:09.836 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:49:09.873 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-16T07:49:09.875 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-16T07:49:09.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:09.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:10.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:10.092 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:10.092 INFO:tasks.ceph:role: osd.2 2024-04-16T07:49:10.092 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:49:10.092 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T07:49:10.159 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:49:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:49:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:49:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:49:10.159 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:49:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:49:10.160 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:49:10.160 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:49:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:49:10.160 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:49:10.196 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-16T07:49:10.197 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-16T07:49:10.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:10.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:10.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:10.655 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:10.656 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-04-16 07:49:10,657 seastar - Reactor backend: linux-aio 2024-04-16T07:49:10.667 INFO:teuthology.orchestra.run.smithi029.stderr:==29464==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x2d8bc 0x29edb 0xdfdb3 0x373a93e 0x5cc1c5e 0x5cc1f0a 0x5cc216a 0x5cc2916 0x5cb31b9 0x5beb14b 0x5bec0f0 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x37d968a 0x37da1e3 0x3679880 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa82628c8 0xffffffffa8265cdb 0xffffffffa7f8a96c 0xffffffffa7c8b731 0xffffffffa8889d52 0xffffffffa8a00b52 2024-04-16T07:49:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:DEBUG 2024-04-16 07:49:10,811 [shard 0:main] seastar - reactor::drain 2024-04-16T07:49:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-04-16T07:49:11.372 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-04-16T07:49:11.372 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:11.372 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-04-16 07:49:11,435 seastar - Reactor backend: linux-aio 2024-04-16T07:49:11.443 INFO:teuthology.orchestra.run.smithi029.stderr:==29462==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:11.508 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x36943 0x5e0d857 0x5d8c2f0 0x5d8c4ca 0x5d8ca56 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:11.508 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:16.111 INFO:teuthology.orchestra.run.smithi029.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 37726edf-a428-419e-bc09-16ace721206f 2024-04-16T07:49:16.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:16.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-04-16 07:49:16,469 seastar - Reactor backend: linux-aio 2024-04-16T07:49:16.473 INFO:teuthology.orchestra.run.smithi029.stderr:==29646==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x172a7 0x174bb 0x2eab 0x4a09d 0x35b7c 0x3629e 0x5313bc4 0x5d89169 0x5d8c554 0x5d8ca56 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x37d968a 0x37da1e3 0x3679880 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-04-16T07:49:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:DEBUG 2024-04-16 07:49:16,617 [shard 0:main] seastar - reactor::drain 2024-04-16T07:49:17.164 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:17.164 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-04-16T07:49:17.164 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-04-16T07:49:17.164 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:17.164 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:17.164 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-04-16 07:49:17,230 seastar - Reactor backend: linux-aio 2024-04-16T07:49:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:==29644==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x36931 0x36900ee 0x5db15e8 0x5db17f1 0x5e0d774 0x5d8c2f0 0x5d8c824 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-04-16T07:49:21.934 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:21.935 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-04-16T07:49:21.935 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-04-16T07:49:21.935 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:21.935 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:21.935 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:21.935 INFO:teuthology.orchestra.run.smithi029.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 37726edf-a428-419e-bc09-16ace721206f 2024-04-16T07:49:21.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:21.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-04-16 07:49:22,301 seastar - Reactor backend: linux-aio 2024-04-16T07:49:22.305 INFO:teuthology.orchestra.run.smithi029.stderr:==29828==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x5d59c13 0x5dbe946 0x5dbeaed 0x5dbedc3 0x5e0d603 0x5e0d809 0x5d8c2f0 0x5d8c824 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x37d968a 0x37da1e3 0x3679880 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-04-16T07:49:22.449 INFO:teuthology.orchestra.run.smithi029.stderr:DEBUG 2024-04-16 07:49:22,450 [shard 0:main] seastar - reactor::drain 2024-04-16T07:49:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-04-16T07:49:22.998 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-04-16T07:49:22.998 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:22.998 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:23.060 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-04-16 07:49:23,062 seastar - Reactor backend: linux-aio 2024-04-16T07:49:23.070 INFO:teuthology.orchestra.run.smithi029.stderr:==29826==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:23.135 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x5339a6a 0x5dc6f8d 0x5dc7039 0x5e0d421 0x5e0d629 0x5e0d809 0x5d8c2f0 0x5d8c4ca 0x5d8ca56 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:23.135 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-04-16T07:49:27.751 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:27.752 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-04-16T07:49:27.752 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-04-16T07:49:27.752 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:27.752 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-04-16T07:49:27.752 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:27.752 INFO:teuthology.orchestra.run.smithi029.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 37726edf-a428-419e-bc09-16ace721206f 2024-04-16T07:49:27.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:27.803 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T07:49:27.803 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:27.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T07:49:27.870 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:27.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T07:49:27.938 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:27.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T07:49:28.007 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:28.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T07:49:28.074 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:28.075 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T07:49:28.130 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T07:49:28.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T07:49:28.196 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2024-04-16T07:49:28.196 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1LR5mk9s5FRAAMiFSYXHqyAphj6tIRUFupA== 2024-04-16T07:49:28.196 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-04-16T07:49:28.196 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB3LR5moVTtIxAA5U3HXhX8jUniaEEPS963fw== 2024-04-16T07:49:28.196 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-04-16T07:49:28.197 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB9LR5mvTu7FxAAMo6Do00mI4VpcSMWne1oOw== 2024-04-16T07:49:28.197 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-04-16T07:49:28.197 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCDLR5mbEriDRAA1QQrF8eTb5Vvy4y/wqsWcQ== 2024-04-16T07:49:28.197 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-04-16T07:49:28.197 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1LR5mWWl1GBAAlU1YQvMrvYWCP/0ggo2NEg== 2024-04-16T07:49:28.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T07:49:28.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:49:28.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:49:28.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:49:28.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:49:28.557 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T07:49:28.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T07:49:28.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T07:49:28.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:28.705+0000 7f9372830a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:28.705+0000 7f9372830a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T07:49:28.746 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:28.801 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T07:49:28.801 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T07:49:28.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T07:49:28.843 INFO:tasks.ceph.mon.a:Started 2024-04-16T07:49:28.844 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T07:49:28.844 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T07:49:28.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T07:49:28.846 INFO:tasks.ceph.mgr.x:Started 2024-04-16T07:49:28.846 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T07:49:28.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T07:49:28.932 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-04-16T07:49:28.932+0000 7f7b8a863a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.932 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-04-16T07:49:28.932+0000 7f7b8a863a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.932 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-04-16T07:49:28.932+0000 7f7b8a863a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.933 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:28.933+0000 7f2d0c97f000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:28.957+0000 7f05e1f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:28.964+0000 7f05e1f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-04-16T07:49:29.047 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T07:49:29.047 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T07:49:29.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T07:49:29.137 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:29.136+0000 7f2d0c97f000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:29.137 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:29.137+0000 7f2d0c97f000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:29.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:29.155+0000 7f2bcd39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:29.162+0000 7f2bcd39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:29.244 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-04-16T07:49:29.244+0000 7f7b815f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T07:49:29.244 INFO:tasks.ceph.mon.a.smithi029.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T07:49:29.244 INFO:tasks.ceph.mon.a.smithi029.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T07:49:29.244 INFO:tasks.ceph.mon.a.smithi029.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T07:49:29.244 INFO:tasks.ceph.mon.a.smithi029.stderr:feature with important data. 2024-04-16T07:49:29.244 INFO:tasks.ceph.mon.a.smithi029.stderr: 2024-04-16T07:49:29.630 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:29.629+0000 7f2d0c97f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T07:49:29.685 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:29.685+0000 7f2d0c97f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T07:49:30.048 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T07:49:30.048 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:30.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T07:49:30.075 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:30.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T07:49:30.144 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:30.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T07:49:30.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd new 4e9b9894-e3e9-425c-a225-d5e55cdaee31 0 2024-04-16T07:49:30.265 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.265+0000 7f2d0c97f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T07:49:30.323 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.323+0000 7f2d0c97f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T07:49:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:30.361+0000 7f4d3c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:30.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:30.368+0000 7f4d3c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:30.372 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.372+0000 7f2d0c97f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T07:49:30.428 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.428+0000 7f2d0c97f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T07:49:30.447 INFO:teuthology.orchestra.run.smithi029.stdout:0 2024-04-16T07:49:30.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd new e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892 1 2024-04-16T07:49:30.474 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.474+0000 7f2d0c97f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T07:49:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:30.583+0000 7f5d5755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:30.590+0000 7f5d5755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:30.604 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.604+0000 7f2d0c97f000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T07:49:30.671 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-04-16T07:49:30.678 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.678+0000 7f2d0c97f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T07:49:30.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd new 18f4917d-4338-46c7-9a94-d22d7c7aff83 2 2024-04-16T07:49:30.728 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.728+0000 7f2d0c97f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T07:49:30.785 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.785+0000 7f2d0c97f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T07:49:30.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:30.791+0000 7f6f09b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:30.799+0000 7f6f09b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:30.836 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:30.836+0000 7f2d0c97f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T07:49:30.877 INFO:teuthology.orchestra.run.smithi029.stdout:2 2024-04-16T07:49:30.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T07:49:30.922 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:49:30.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:49:30.964 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:49:30.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T07:49:30.998 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:49:30.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:49:31.042 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:49:31.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T07:49:31.077 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T07:49:31.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T07:49:31.137 INFO:tasks.ceph.osd.2:Started 2024-04-16T07:49:31.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:31.264+0000 7f6a0235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:31.271+0000 7f6a0235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:31.277 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:31.277+0000 7f2d0c97f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T07:49:31.355 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:31.355 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"fsid":"37726edf-a428-419e-bc09-16ace721206f","created":"2024-04-16T07:49:28.951424+0000","modified":"2024-04-16T07:49:30.878451+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:49:30.878451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4e9b9894-e3e9-425c-a225-d5e55cdaee31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18f4917d-4338-46c7-9a94-d22d7c7aff83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:31.368 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T07:49:31.368 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T07:49:31.514 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:31.513+0000 7f2d0c97f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T07:49:31.670 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:31.726 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:31.726+0000 7f2d0c97f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T07:49:31.773 INFO:teuthology.misc.health.smithi029.stderr:2024-04-16T07:49:31.771+0000 7f4351c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:31.776 INFO:teuthology.misc.health.smithi029.stderr:2024-04-16T07:49:31.776+0000 7f4351c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:31.811 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:31.811+0000 7f2d0c97f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T07:49:31.842 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-16T07:49:31.842 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":6,"fsid":"37726edf-a428-419e-bc09-16ace721206f","created":"2024-04-16T07:49:28.951424+0000","modified":"2024-04-16T07:49:30.878451+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:49:30.878451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4e9b9894-e3e9-425c-a225-d5e55cdaee31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18f4917d-4338-46c7-9a94-d22d7c7aff83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:31.853 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-16T07:49:32.026 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:32.025+0000 7f2d0c97f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:49:32.195 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:32.195+0000 7f2d0c97f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T07:49:32.475 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:32.476+0000 7f2d0c97f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:49:32.549 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:32.550+0000 7f2d0c97f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T07:49:32.832 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:32.833+0000 7f2d0c97f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T07:49:34.027 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:34.028+0000 7f2d0c97f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T07:49:34.326 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:34.327+0000 7f2d0c97f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T07:49:34.389 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:34.389+0000 7f2d0c97f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T07:49:34.524 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:34.524+0000 7f2d0c97f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T07:49:35.533 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-16T07:49:35.534+0000 7f2cabbfc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T07:49:38.156 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:38.272 INFO:teuthology.misc.health.smithi029.stderr:2024-04-16T07:49:38.270+0000 7f29f52ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.274 INFO:teuthology.misc.health.smithi029.stderr:2024-04-16T07:49:38.275+0000 7f29f52ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.470 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-16T07:49:38.470 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":8,"fsid":"37726edf-a428-419e-bc09-16ace721206f","created":"2024-04-16T07:49:28.951424+0000","modified":"2024-04-16T07:49:37.534890+0000","last_up_change":"2024-04-16T07:49:37.534890+0000","last_in_change":"2024-04-16T07:49:30.878451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4e9b9894-e3e9-425c-a225-d5e55cdaee31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":35903630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":35903630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":35903630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":35903630}]},"public_addr":"172.21.15.29:6809/35903630","cluster_addr":"172.21.15.29:6808/35903630","heartbeat_back_addr":"172.21.15.29:6812/35903630","heartbeat_front_addr":"172.21.15.29:6811/35903630","state":["exists","up"]},{"osd":1,"uuid":"e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":3733026672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3733026672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3733026672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3733026672}]},"public_addr":"172.21.15.29:6805/3733026672","cluster_addr":"172.21.15.29:6804/3733026672","heartbeat_back_addr":"172.21.15.29:6806/3733026672","heartbeat_front_addr":"172.21.15.29:6802/3733026672","state":["exists","up"]},{"osd":2,"uuid":"18f4917d-4338-46c7-9a94-d22d7c7aff83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:38.482 DEBUG:teuthology.misc:2 of 3 OSDs are up 2024-04-16T07:49:44.785 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:44.902 INFO:teuthology.misc.health.smithi029.stderr:2024-04-16T07:49:44.900+0000 7f07f146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:44.904 INFO:teuthology.misc.health.smithi029.stderr:2024-04-16T07:49:44.906+0000 7f07f146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:45.101 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-16T07:49:45.101 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":12,"fsid":"37726edf-a428-419e-bc09-16ace721206f","created":"2024-04-16T07:49:28.951424+0000","modified":"2024-04-16T07:49:41.548897+0000","last_up_change":"2024-04-16T07:49:38.538244+0000","last_in_change":"2024-04-16T07:49:30.878451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:49:38.547254+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e9b9894-e3e9-425c-a225-d5e55cdaee31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":35903630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":35903630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":35903630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":35903630}]},"public_addr":"172.21.15.29:6809/35903630","cluster_addr":"172.21.15.29:6808/35903630","heartbeat_back_addr":"172.21.15.29:6812/35903630","heartbeat_front_addr":"172.21.15.29:6811/35903630","state":["exists","up"]},{"osd":1,"uuid":"e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":3733026672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3733026672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3733026672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3733026672}]},"public_addr":"172.21.15.29:6805/3733026672","cluster_addr":"172.21.15.29:6804/3733026672","heartbeat_back_addr":"172.21.15.29:6806/3733026672","heartbeat_front_addr":"172.21.15.29:6802/3733026672","state":["exists","up"]},{"osd":2,"uuid":"18f4917d-4338-46c7-9a94-d22d7c7aff83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1261843266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1261843266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1261843266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1261843266}]},"public_addr":"172.21.15.29:6813/1261843266","cluster_addr":"172.21.15.29:6810/1261843266","heartbeat_back_addr":"172.21.15.29:6815/1261843266","heartbeat_front_addr":"172.21.15.29:6814/1261843266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:45.112 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-16T07:49:45.113 INFO:tasks.ceph:Creating RBD pool 2024-04-16T07:49:45.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T07:49:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:45.225+0000 7f987315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:45.233+0000 7f987315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:46.156 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2024-04-16T07:49:46.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T07:49:46.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:46.312+0000 7ff042e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:46.323+0000 7ff042e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T07:49:47.181 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T07:49:47.181 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T07:49:47.181 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T07:49:47.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T07:49:47.182 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T07:49:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:47.338+0000 7f875b75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:47.344+0000 7f875b75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:47.593 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:47.604 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3935438064},{"type":"v1","addr":"172.21.15.29:6801","nonce":3935438064}]},"active_addr":"172.21.15.29:6801/3935438064","active_change":"2024-04-16T07:49:34.526637+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":4030908027}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1372773642}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":957799205}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2245919318}]}]} 2024-04-16T07:49:47.608 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T07:49:47.608 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:49:47.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:47.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:47.721+0000 7fa7978ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:47.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:47.726+0000 7fa7978ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:47.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:47.924 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":14,"fsid":"37726edf-a428-419e-bc09-16ace721206f","created":"2024-04-16T07:49:28.951424+0000","modified":"2024-04-16T07:49:47.157444+0000","last_up_change":"2024-04-16T07:49:38.538244+0000","last_in_change":"2024-04-16T07:49:30.878451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:49:38.547254+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:49:45.431470+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e9b9894-e3e9-425c-a225-d5e55cdaee31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":35903630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":35903630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":35903630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":35903630}]},"public_addr":"172.21.15.29:6809/35903630","cluster_addr":"172.21.15.29:6808/35903630","heartbeat_back_addr":"172.21.15.29:6812/35903630","heartbeat_front_addr":"172.21.15.29:6811/35903630","state":["exists","up"]},{"osd":1,"uuid":"e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":3733026672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3733026672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3733026672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3733026672}]},"public_addr":"172.21.15.29:6805/3733026672","cluster_addr":"172.21.15.29:6804/3733026672","heartbeat_back_addr":"172.21.15.29:6806/3733026672","heartbeat_front_addr":"172.21.15.29:6802/3733026672","state":["exists","up"]},{"osd":2,"uuid":"18f4917d-4338-46c7-9a94-d22d7c7aff83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1261843266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1261843266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1261843266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1261843266}]},"public_addr":"172.21.15.29:6813/1261843266","cluster_addr":"172.21.15.29:6810/1261843266","heartbeat_back_addr":"172.21.15.29:6815/1261843266","heartbeat_front_addr":"172.21.15.29:6814/1261843266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:47.937 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T07:49:47.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:48.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.050+0000 7f84ce664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.056+0000 7f84ce664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:48.251 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"fsid":"37726edf-a428-419e-bc09-16ace721206f","created":"2024-04-16T07:49:28.951424+0000","modified":"2024-04-16T07:49:48.160516+0000","last_up_change":"2024-04-16T07:49:38.538244+0000","last_in_change":"2024-04-16T07:49:30.878451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:49:38.547254+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:49:45.431470+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e9b9894-e3e9-425c-a225-d5e55cdaee31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":35903630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":35903630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":35903630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":35903630}]},"public_addr":"172.21.15.29:6809/35903630","cluster_addr":"172.21.15.29:6808/35903630","heartbeat_back_addr":"172.21.15.29:6812/35903630","heartbeat_front_addr":"172.21.15.29:6811/35903630","state":["exists","up"]},{"osd":1,"uuid":"e36d27c7-5b0c-4a2f-9a0d-dcdfe8f17892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":3733026672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3733026672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3733026672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3733026672}]},"public_addr":"172.21.15.29:6805/3733026672","cluster_addr":"172.21.15.29:6804/3733026672","heartbeat_back_addr":"172.21.15.29:6806/3733026672","heartbeat_front_addr":"172.21.15.29:6802/3733026672","state":["exists","up"]},{"osd":2,"uuid":"18f4917d-4338-46c7-9a94-d22d7c7aff83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1261843266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1261843266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1261843266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1261843266}]},"public_addr":"172.21.15.29:6813/1261843266","cluster_addr":"172.21.15.29:6810/1261843266","heartbeat_back_addr":"172.21.15.29:6815/1261843266","heartbeat_front_addr":"172.21.15.29:6814/1261843266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:48.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T07:49:48.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T07:49:48.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T07:49:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.383+0000 7f1094c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.383+0000 7f36242ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.383+0000 7f5223b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.390+0000 7f36242ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.390+0000 7f1094c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.391+0000 7f5223b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:34359738378 2024-04-16T07:49:48.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:49:48.423 INFO:teuthology.orchestra.run.smithi029.stdout:38654705673 2024-04-16T07:49:48.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:49:48.425 INFO:teuthology.orchestra.run.smithi029.stdout:34359738378 2024-04-16T07:49:48.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:49:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.536+0000 7feea0c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.543+0000 7feea0c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.592+0000 7f1932c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.599+0000 7f1932c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.594+0000 7f657f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:48.602+0000 7f657f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:48.758 INFO:teuthology.orchestra.run.smithi029.stdout:34359738378 2024-04-16T07:49:48.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738378 got 34359738378 for osd.1 2024-04-16T07:49:48.771 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:48.822 INFO:teuthology.orchestra.run.smithi029.stdout:38654705669 2024-04-16T07:49:48.825 INFO:teuthology.orchestra.run.smithi029.stdout:34359738377 2024-04-16T07:49:48.835 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705669 for osd.2 2024-04-16T07:49:48.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738378 got 34359738377 for osd.0 2024-04-16T07:49:49.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:49:49.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:49:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:49.962+0000 7ff31115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:49.962+0000 7fcebd6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:49.969+0000 7ff31115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:49.969+0000 7fcebd6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.167 INFO:teuthology.orchestra.run.smithi029.stdout:38654705673 2024-04-16T07:49:50.171 INFO:teuthology.orchestra.run.smithi029.stdout:34359738378 2024-04-16T07:49:50.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.2 2024-04-16T07:49:50.181 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:50.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738378 got 34359738378 for osd.0 2024-04-16T07:49:50.185 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:50.185 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:49:50.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:50.340+0000 7f532119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:50.345+0000 7f532119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-16T07:49:50.548 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T07:49:50.531810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80940,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034580,"statfs":{"total":289910292480,"available":289827409920,"internally_reserved":0,"allocated":479232,"data_stored":1264856,"data_compressed":59820,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.956799"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.279218+0000","last_change":"2024-04-16T07:49:47.279388+0000","last_active":"2024-04-16T07:49:47.279218+0000","last_peered":"2024-04-16T07:49:47.279218+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.277044+0000","last_became_peered":"2024-04-16T07:49:47.277044+0000","last_unstale":"2024-04-16T07:49:47.279218+0000","last_undegraded":"2024-04-16T07:49:47.279218+0000","last_fullsized":"2024-04-16T07:49:47.279218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.265649+0000","last_change":"2024-04-16T07:49:47.265981+0000","last_active":"2024-04-16T07:49:47.265649+0000","last_peered":"2024-04-16T07:49:47.265649+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.261768+0000","last_became_peered":"2024-04-16T07:49:47.261768+0000","last_unstale":"2024-04-16T07:49:47.265649+0000","last_undegraded":"2024-04-16T07:49:47.265649+0000","last_fullsized":"2024-04-16T07:49:47.265649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.242266+0000","last_change":"2024-04-16T07:49:47.242833+0000","last_active":"2024-04-16T07:49:47.242266+0000","last_peered":"2024-04-16T07:49:47.242266+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.238299+0000","last_became_peered":"2024-04-16T07:49:47.238299+0000","last_unstale":"2024-04-16T07:49:47.242266+0000","last_undegraded":"2024-04-16T07:49:47.242266+0000","last_fullsized":"2024-04-16T07:49:47.242266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.279994+0000","last_change":"2024-04-16T07:49:47.280157+0000","last_active":"2024-04-16T07:49:47.279994+0000","last_peered":"2024-04-16T07:49:47.279994+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.278219+0000","last_became_peered":"2024-04-16T07:49:47.278219+0000","last_unstale":"2024-04-16T07:49:47.279994+0000","last_undegraded":"2024-04-16T07:49:47.279994+0000","last_fullsized":"2024-04-16T07:49:47.279994+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T07:49:48.172132+0000","last_change":"2024-04-16T07:49:47.239625+0000","last_active":"2024-04-16T07:49:48.172132+0000","last_peered":"2024-04-16T07:49:48.172132+0000","last_clean":"2024-04-16T07:49:48.172132+0000","last_became_active":"2024-04-16T07:49:47.237551+0000","last_became_peered":"2024-04-16T07:49:47.237551+0000","last_unstale":"2024-04-16T07:49:48.172132+0000","last_undegraded":"2024-04-16T07:49:48.172132+0000","last_fullsized":"2024-04-16T07:49:48.172132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.235238+0000","last_change":"2024-04-16T07:49:47.235591+0000","last_active":"2024-04-16T07:49:47.235238+0000","last_peered":"2024-04-16T07:49:47.235238+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.231874+0000","last_became_peered":"2024-04-16T07:49:47.231874+0000","last_unstale":"2024-04-16T07:49:47.235238+0000","last_undegraded":"2024-04-16T07:49:47.235238+0000","last_fullsized":"2024-04-16T07:49:47.235238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.226670+0000","last_change":"2024-04-16T07:49:47.226994+0000","last_active":"2024-04-16T07:49:47.226670+0000","last_peered":"2024-04-16T07:49:47.226670+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.223908+0000","last_became_peered":"2024-04-16T07:49:47.223908+0000","last_unstale":"2024-04-16T07:49:47.226670+0000","last_undegraded":"2024-04-16T07:49:47.226670+0000","last_fullsized":"2024-04-16T07:49:47.226670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.216502+0000","last_change":"2024-04-16T07:49:47.216809+0000","last_active":"2024-04-16T07:49:47.216502+0000","last_peered":"2024-04-16T07:49:47.216502+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.214304+0000","last_became_peered":"2024-04-16T07:49:47.214304+0000","last_unstale":"2024-04-16T07:49:47.216502+0000","last_undegraded":"2024-04-16T07:49:47.216502+0000","last_fullsized":"2024-04-16T07:49:47.216502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.174413+0000","last_change":"2024-04-16T07:49:40.583980+0000","last_active":"2024-04-16T07:49:47.174413+0000","last_peered":"2024-04-16T07:49:47.174413+0000","last_clean":"2024-04-16T07:49:47.174413+0000","last_became_active":"2024-04-16T07:49:40.581306+0000","last_became_peered":"2024-04-16T07:49:40.581306+0000","last_unstale":"2024-04-16T07:49:47.174413+0000","last_undegraded":"2024-04-16T07:49:47.174413+0000","last_fullsized":"2024-04-16T07:49:47.174413+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:39.541702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:39.541702+0000","last_clean_scrub_stamp":"2024-04-16T07:49:39.541702+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618408,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618408,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":28040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:50.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:50.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:50.663+0000 7f2ab9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:50.668+0000 7f2ab9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.859 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:50.859 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-16T07:49:50.873 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T07:49:50.531810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80940,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034580,"statfs":{"total":289910292480,"available":289827409920,"internally_reserved":0,"allocated":479232,"data_stored":1264856,"data_compressed":59820,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.956799"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.279218+0000","last_change":"2024-04-16T07:49:47.279388+0000","last_active":"2024-04-16T07:49:47.279218+0000","last_peered":"2024-04-16T07:49:47.279218+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.277044+0000","last_became_peered":"2024-04-16T07:49:47.277044+0000","last_unstale":"2024-04-16T07:49:47.279218+0000","last_undegraded":"2024-04-16T07:49:47.279218+0000","last_fullsized":"2024-04-16T07:49:47.279218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.265649+0000","last_change":"2024-04-16T07:49:47.265981+0000","last_active":"2024-04-16T07:49:47.265649+0000","last_peered":"2024-04-16T07:49:47.265649+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.261768+0000","last_became_peered":"2024-04-16T07:49:47.261768+0000","last_unstale":"2024-04-16T07:49:47.265649+0000","last_undegraded":"2024-04-16T07:49:47.265649+0000","last_fullsized":"2024-04-16T07:49:47.265649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.242266+0000","last_change":"2024-04-16T07:49:47.242833+0000","last_active":"2024-04-16T07:49:47.242266+0000","last_peered":"2024-04-16T07:49:47.242266+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.238299+0000","last_became_peered":"2024-04-16T07:49:47.238299+0000","last_unstale":"2024-04-16T07:49:47.242266+0000","last_undegraded":"2024-04-16T07:49:47.242266+0000","last_fullsized":"2024-04-16T07:49:47.242266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.279994+0000","last_change":"2024-04-16T07:49:47.280157+0000","last_active":"2024-04-16T07:49:47.279994+0000","last_peered":"2024-04-16T07:49:47.279994+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.278219+0000","last_became_peered":"2024-04-16T07:49:47.278219+0000","last_unstale":"2024-04-16T07:49:47.279994+0000","last_undegraded":"2024-04-16T07:49:47.279994+0000","last_fullsized":"2024-04-16T07:49:47.279994+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T07:49:48.172132+0000","last_change":"2024-04-16T07:49:47.239625+0000","last_active":"2024-04-16T07:49:48.172132+0000","last_peered":"2024-04-16T07:49:48.172132+0000","last_clean":"2024-04-16T07:49:48.172132+0000","last_became_active":"2024-04-16T07:49:47.237551+0000","last_became_peered":"2024-04-16T07:49:47.237551+0000","last_unstale":"2024-04-16T07:49:48.172132+0000","last_undegraded":"2024-04-16T07:49:48.172132+0000","last_fullsized":"2024-04-16T07:49:48.172132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.235238+0000","last_change":"2024-04-16T07:49:47.235591+0000","last_active":"2024-04-16T07:49:47.235238+0000","last_peered":"2024-04-16T07:49:47.235238+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.231874+0000","last_became_peered":"2024-04-16T07:49:47.231874+0000","last_unstale":"2024-04-16T07:49:47.235238+0000","last_undegraded":"2024-04-16T07:49:47.235238+0000","last_fullsized":"2024-04-16T07:49:47.235238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.226670+0000","last_change":"2024-04-16T07:49:47.226994+0000","last_active":"2024-04-16T07:49:47.226670+0000","last_peered":"2024-04-16T07:49:47.226670+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.223908+0000","last_became_peered":"2024-04-16T07:49:47.223908+0000","last_unstale":"2024-04-16T07:49:47.226670+0000","last_undegraded":"2024-04-16T07:49:47.226670+0000","last_fullsized":"2024-04-16T07:49:47.226670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.216502+0000","last_change":"2024-04-16T07:49:47.216809+0000","last_active":"2024-04-16T07:49:47.216502+0000","last_peered":"2024-04-16T07:49:47.216502+0000","last_clean":"2024-04-16T07:49:46.153584+0000","last_became_active":"2024-04-16T07:49:47.214304+0000","last_became_peered":"2024-04-16T07:49:47.214304+0000","last_unstale":"2024-04-16T07:49:47.216502+0000","last_undegraded":"2024-04-16T07:49:47.216502+0000","last_fullsized":"2024-04-16T07:49:47.216502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:46.153584+0000","last_clean_scrub_stamp":"2024-04-16T07:49:46.153584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:49:47.174413+0000","last_change":"2024-04-16T07:49:40.583980+0000","last_active":"2024-04-16T07:49:47.174413+0000","last_peered":"2024-04-16T07:49:47.174413+0000","last_clean":"2024-04-16T07:49:47.174413+0000","last_became_active":"2024-04-16T07:49:40.581306+0000","last_became_peered":"2024-04-16T07:49:40.581306+0000","last_unstale":"2024-04-16T07:49:47.174413+0000","last_undegraded":"2024-04-16T07:49:47.174413+0000","last_fullsized":"2024-04-16T07:49:47.174413+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:39.541702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:39.541702+0000","last_clean_scrub_stamp":"2024-04-16T07:49:39.541702+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618408,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618408,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":28040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:50.874 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:49:50.874 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T07:49:50.874 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T07:49:50.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T07:49:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:50.988+0000 7fd2dfb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:49:50.992+0000 7fd2dfb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:51.227 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:51.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T07:49:51.240 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T07:49:51.240 INFO:teuthology.run_tasks:Running task cbt... 2024-04-16T07:49:51.255 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi029.front.sepia.ceph.com', 'osds': ['ubuntu@smithi029.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi029.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi029.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi029.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-16T07:49:51.255 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-16T07:49:51.274 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:49:51.274 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-16T07:49:51.326 DEBUG:teuthology.orchestra.run.smithi029:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-16T07:49:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-16T07:49:51.827 INFO:tasks.cbt:Installing collectl 2024-04-16T07:49:51.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-16T07:49:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:--2024-04-16 07:49:51-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-16T07:49:52.108 INFO:teuthology.orchestra.run.smithi029.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2024-04-16T07:49:52.118 INFO:teuthology.orchestra.run.smithi029.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-04-16T07:49:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-16T07:49:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmHi2geFayj-cKRn5F7M1V2tBWbDN44UeZjUoUVpqC1lEo_aC3nBDchz1fuWrv6p4bIv7-zY2SI_Q4Ihks7_YvIhgpeg%3D%3D&use_mirror=cytranet&r= [following] 2024-04-16T07:49:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:--2024-04-16 07:49:52-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmHi2geFayj-cKRn5F7M1V2tBWbDN44UeZjUoUVpqC1lEo_aC3nBDchz1fuWrv6p4bIv7-zY2SI_Q4Ihks7_YvIhgpeg%3D%3D&use_mirror=cytranet&r= 2024-04-16T07:49:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-16T07:49:52.465 INFO:teuthology.orchestra.run.smithi029.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-16T07:49:52.748 INFO:teuthology.orchestra.run.smithi029.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-16T07:49:52.748 INFO:teuthology.orchestra.run.smithi029.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-16T07:49:52.748 INFO:teuthology.orchestra.run.smithi029.stderr:--2024-04-16 07:49:52-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-16T07:49:52.762 INFO:teuthology.orchestra.run.smithi029.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2024-04-16T07:49:52.832 INFO:teuthology.orchestra.run.smithi029.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2024-04-16T07:49:53.155 INFO:teuthology.orchestra.run.smithi029.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-16T07:49:53.155 INFO:teuthology.orchestra.run.smithi029.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-16T07:49:53.155 INFO:teuthology.orchestra.run.smithi029.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-16T07:49:53.155 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:53.238 INFO:teuthology.orchestra.run.smithi029.stderr: 0K .......... .......... .......... .......... .......... 8% 602K 1s 2024-04-16T07:49:53.308 INFO:teuthology.orchestra.run.smithi029.stderr: 50K .......... .......... .......... .......... .......... 16% 719K 1s 2024-04-16T07:49:53.308 INFO:teuthology.orchestra.run.smithi029.stderr: 100K .......... .......... .......... .......... .......... 24% 47.2M 0s 2024-04-16T07:49:53.364 INFO:teuthology.orchestra.run.smithi029.stderr: 150K .......... .......... .......... .......... .......... 32% 893K 0s 2024-04-16T07:49:53.379 INFO:teuthology.orchestra.run.smithi029.stderr: 200K .......... .......... .......... .......... .......... 41% 3.61M 0s 2024-04-16T07:49:53.379 INFO:teuthology.orchestra.run.smithi029.stderr: 250K .......... .......... .......... .......... .......... 49% 62.6M 0s 2024-04-16T07:49:53.380 INFO:teuthology.orchestra.run.smithi029.stderr: 300K .......... .......... .......... .......... .......... 57% 59.1M 0s 2024-04-16T07:49:53.434 INFO:teuthology.orchestra.run.smithi029.stderr: 350K .......... .......... .......... .......... .......... 65% 914K 0s 2024-04-16T07:49:53.447 INFO:teuthology.orchestra.run.smithi029.stderr: 400K .......... .......... .......... .......... .......... 74% 3.88M 0s 2024-04-16T07:49:53.448 INFO:teuthology.orchestra.run.smithi029.stderr: 450K .......... .......... .......... .......... .......... 82% 74.8M 0s 2024-04-16T07:49:53.450 INFO:teuthology.orchestra.run.smithi029.stderr: 500K .......... .......... .......... .......... .......... 90% 37.6M 0s 2024-04-16T07:49:53.450 INFO:teuthology.orchestra.run.smithi029.stderr: 550K .......... .......... .......... .......... .......... 98% 193M 0s 2024-04-16T07:49:53.450 INFO:teuthology.orchestra.run.smithi029.stderr: 600K ...... 100% 11.4T=0.3s 2024-04-16T07:49:53.450 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16 07:49:53 (2.01 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-16T07:49:53.450 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:49:53.494 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/ 2024-04-16T07:49:53.494 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/collectl.conf 2024-04-16T07:49:53.494 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/ 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Socket.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/colmux.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Import.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/style.css 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-16T07:49:53.495 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Process.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Export.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Startup.html 2024-04-16T07:49:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Performance.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Messages.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Logging.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Misc.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Examples.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-16T07:49:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Playback.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Hello.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Network.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Memory.html 2024-04-16T07:49:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Disks.html 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/service/ 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/service/collectl.service 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/ 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/misc.ph 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/gexpr.ph 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/statsd.ph 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/envrules.std 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/hello.ph 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/vmsum.ph 2024-04-16T07:49:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/proctree.ph 2024-04-16T07:49:53.500 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/README-WINDOWS 2024-04-16T07:49:53.500 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/formatit.ph 2024-04-16T07:49:53.501 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/colmux 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/INSTALL 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/graphite.ph 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/vmstat.ph 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/README 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/man1/ 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/man1/collectl.1 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/man1/colmux.1 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/lexpr.ph 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/client.pl 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/collectl 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/COPYING 2024-04-16T07:49:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/vnet.ph 2024-04-16T07:49:53.503 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/GPL 2024-04-16T07:49:53.503 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/UNINSTALL 2024-04-16T07:49:53.503 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/ARTISTIC 2024-04-16T07:49:53.555 INFO:teuthology.orchestra.run.smithi029.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-16T07:49:53.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-16T07:49:54.150 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:08:29 ago on Tue 16 Apr 2024 07:41:25 AM UTC. 2024-04-16T07:49:54.254 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-16T07:49:54.256 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-16T07:49:54.317 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout: librbd-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 23 k 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout: perf x86_64 5.14.0-438.el9 CentOS-AppStream 2.2 M 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-16T07:49:54.318 INFO:teuthology.orchestra.run.smithi029.stdout: libradospp-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 30 k 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:Install 5 Packages 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.4 M 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 11 M 2024-04-16T07:49:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-16T07:49:54.906 INFO:teuthology.orchestra.run.smithi029.stdout:(1/5): librbd-devel-19.0.0-3109.g30b0aa00.el9.x 113 kB/s | 23 kB 00:00 2024-04-16T07:49:55.090 INFO:teuthology.orchestra.run.smithi029.stdout:(2/5): perf-5.14.0-438.el9.x86_64.rpm 5.8 MB/s | 2.2 MB 00:00 2024-04-16T07:49:55.392 INFO:teuthology.orchestra.run.smithi029.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 44 kB/s | 13 kB 00:00 2024-04-16T07:49:55.426 INFO:teuthology.orchestra.run.smithi029.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 228 kB/s | 118 kB 00:00 2024-04-16T07:49:55.476 INFO:teuthology.orchestra.run.smithi029.stdout:(5/5): libradospp-devel-19.0.0-3109.g30b0aa00.e 39 kB/s | 30 kB 00:00 2024-04-16T07:49:55.480 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:49:55.480 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 2.4 MB 00:01 2024-04-16T07:49:55.480 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-16T07:49:55.500 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-16T07:49:55.500 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-16T07:49:55.634 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-16T07:49:55.634 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-16T07:49:55.921 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-16T07:49:55.996 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-16T07:49:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-16T07:49:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradospp-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/5 2024-04-16T07:49:56.406 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librbd-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/5 2024-04-16T07:49:56.504 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : perf-5.14.0-438.el9.x86_64 5/5 2024-04-16T07:49:57.662 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: perf-5.14.0-438.el9.x86_64 5/5 2024-04-16T07:49:57.662 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradospp-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/5 2024-04-16T07:49:57.662 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/5 2024-04-16T07:49:57.662 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : perf-5.14.0-438.el9.x86_64 3/5 2024-04-16T07:49:57.662 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-16T07:49:58.027 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-16T07:49:58.027 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:58.027 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-16T07:49:58.027 INFO:teuthology.orchestra.run.smithi029.stdout: libradospp-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:49:58.028 INFO:teuthology.orchestra.run.smithi029.stdout: librbd-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:49:58.028 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh-2.34-7.el9.x86_64 2024-04-16T07:49:58.028 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-16T07:49:58.028 INFO:teuthology.orchestra.run.smithi029.stdout: perf-5.14.0-438.el9.x86_64 2024-04-16T07:49:58.028 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-16T07:49:58.028 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-16T07:49:58.466 INFO:tasks.cbt:benchmark: radosbench 2024-04-16T07:49:58.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-16T07:49:58.496 DEBUG:teuthology.orchestra.run.smithi029:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - Settings.cluster: 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'clients': ['ubuntu@smithi029.front.sepia.ceph.com'], 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'head': 'ubuntu@smithi029.front.sepia.ceph.com', 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'is_teuthology': True, 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'iterations': 1, 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'mons': ['ubuntu@smithi029.front.sepia.ceph.com'], 2024-04-16T07:49:58.944 INFO:teuthology.orchestra.run.smithi029.stderr: 'osds': ['ubuntu@smithi029.front.sepia.ceph.com'], 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'osds_per_node': 3, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'pid_dir': '/var/run/ceph', 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'pgp_size': 256, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'replication': 'replicated'}}, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'rebuild_every_test': False, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'rgws': {'ubuntu@smithi029.front.sepia.ceph.com': None}, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'use_existing': True, 2024-04-16T07:49:58.945 INFO:teuthology.orchestra.run.smithi029.stderr: 'user': 'ubuntu'} 2024-04-16T07:49:58.965 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-04-16T07:49:58.966 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:49:58.967 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:49:58.967 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-04-16T07:49:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:49:58.971 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-16T07:49:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:49:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:49:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-04-16T07:49:58.975 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:49:58.975 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-16T07:49:58.975 INFO:teuthology.orchestra.run.smithi029.stderr:07:49:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-16T07:50:59.020 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.020 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-16T07:50:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-16T07:50:59.112 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.112 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-04-16T07:50:59.114 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.114 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-04-16T07:50:59.116 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.116 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.116 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-04-16T07:50:59.134 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.134 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/$(basename ${i}).smithi029.front.sepia.ceph.com ; done 2024-04-16T07:50:59.138 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-04-16T07:50:59.170 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - WARNING - cbt - sudo killall -9 rados 2024-04-16T07:50:59.170 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - WARNING - cbt - error 1 seen, continuing anyway... 2024-04-16T07:50:59.190 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d. 2024-04-16T07:50:59.190 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-04-16T07:50:59.190 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.190 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.190 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-04-16T07:50:59.207 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-04-16T07:50:59.210 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.211 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-16T07:50:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-04-16T07:50:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-16T07:50:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-16T07:50:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-04-16T07:50:59.571 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:50:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:07:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-04-16T07:51:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:00 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:00 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-04-16T07:51:01.652 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:01 - INFO - cbt - Checking Health after pool creation. 2024-04-16T07:51:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:06 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:06 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-16T07:51:07.002 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:07 - INFO - cbt - Final Pool Health Check. 2024-04-16T07:51:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-16T07:51:12.345 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.345 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-16T07:51:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-16T07:51:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-04-16T07:51:12.405 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.405 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-04-16T07:51:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-16T07:51:12.604 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:12.604 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-16T07:51:13.170 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:13.170 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:13.170 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-16T07:51:13.667 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:13.667 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:13.667 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-04-16T07:51:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-04-16T07:51:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-04-16T07:51:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-16T07:51:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-04-16T07:51:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-16T07:51:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-16T07:51:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - INFO - cbt - Running radosbench write test. 2024-04-16T07:51:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-04-16T07:51:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:51:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:07:51:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-04-16T07:52:16.033 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.034 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-16T07:52:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.074 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-16T07:52:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-04-16T07:52:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-04-16T07:52:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-04-16T07:52:16.443 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.451 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.451 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-04-16T07:52:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/$(basename ${i}).smithi029.front.sepia.ceph.com ; done 2024-04-16T07:52:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - INFO - cbt - Convert results to json format. 2024-04-16T07:52:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-16T07:52:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-16T07:52:17.452 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:17 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:17 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:17 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq 2024-04-16T07:52:17.527 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:17 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:17.527 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:17 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:17.527 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/ceph_settings.out 2024-04-16T07:52:18.638 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.638 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.639 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/collectl 2024-04-16T07:52:18.641 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.641 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-16T07:52:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.643 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf 2024-04-16T07:52:18.645 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-16T07:52:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-16T07:52:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - INFO - cbt - Running radosbench seq test. 2024-04-16T07:52:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/output.0 2024-04-16T07:52:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/output.1 2024-04-16T07:52:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:35.841 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-16T07:52:35.877 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:35.877 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-16T07:52:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf/perf.data 2024-04-16T07:52:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.* 2024-04-16T07:52:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/historic_ops.out 2024-04-16T07:52:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.236 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/* 2024-04-16T07:52:36.253 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.253 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/$(basename ${i}).smithi029.front.sepia.ceph.com ; done 2024-04-16T07:52:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - INFO - cbt - Convert results to json format. 2024-04-16T07:52:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:07:52:36 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.290 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-16T07:52:36.300 DEBUG:teuthology.orchestra.run.smithi029:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-16T07:52:36.507 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-16T07:52:36.507 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-16T07:52:36.507 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-16T07:52:36.565 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.565 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.565 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.566 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.smithi029.front.sepia.ceph.com 2024-04-16T07:52:36.567 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:52:36.567 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.smithi029.front.sepia.ceph.com of=/dev/stdout 2024-04-16T07:52:36.598 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-16T07:52:36.598 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2024-04-16T07:52:36.653 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-16T07:52:36.653 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-16T07:52:36.653 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:52:36.654 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.smithi029.front.sepia.ceph.com of=/dev/stdout 2024-04-16T07:52:36.709 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-16T07:52:36.709 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2024-04-16T07:52:36.765 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-16T07:52:36.765 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-16T07:52:36.765 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:52:36.765 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.smithi029.front.sepia.ceph.com of=/dev/stdout 2024-04-16T07:52:36.821 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-16T07:52:36.821 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2024-04-16T07:52:36.877 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-16T07:52:36.877 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-16T07:52:36.877 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-16T07:52:36.878 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.smithi029.front.sepia.ceph.com of=/dev/stdout 2024-04-16T07:52:36.933 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-16T07:52:36.933 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2024-04-16T07:52:36.989 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-16T07:52:36.989 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-16T07:52:36.998 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-16T07:52:37.004 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-16T07:52:37.011 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-16T07:52:37.017 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-16T07:52:37.017 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T07:52:37.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-16T07:52:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:52:37.150+0000 7f085b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-16T07:52:37.158+0000 7f085b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:37.372 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T07:52:37.372 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T07:52:37.373 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:52:37.373 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:52:38.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-16T07:52:43.474 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:52:43.475 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:52:43.475 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:52:43.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-16T07:52:48.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-16T07:52:48.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-16T07:52:49.576 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:52:49.577 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T07:52:49.577 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:52:54.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-16T07:52:54.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-16T07:52:55.678 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T07:52:55.678 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T07:52:55.679 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T07:52:55.679 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:52:55.712 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T07:52:55.712 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T07:52:55.713 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T07:52:55.713 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:52:55.766 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T07:52:55.766 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T07:52:55.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-16T07:52:55.868 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:55.869 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T07:52:56.189 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:56.189 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T07:52:56.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:52:56.309 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T07:52:56.375 INFO:tasks.ceph:Archiving mon data... 2024-04-16T07:52:56.376 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001/data/mon.a.tgz 2024-04-16T07:52:56.376 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-04-16T07:52:56.418 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.KY3B14x8d7 2024-04-16T07:52:56.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KY3B14x8d7 2024-04-16T07:52:56.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.KY3B14x8d7 2024-04-16T07:52:56.660 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.KY3B14x8d7 is 186KB 2024-04-16T07:52:56.713 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.KY3B14x8d7 2024-04-16T07:52:56.728 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T07:52:56.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T07:52:56.858 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T07:52:56.861 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T07:52:56.863 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001/remote/smithi029/crash 2024-04-16T07:52:56.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T07:52:56.925 INFO:tasks.ceph:Compressing logs... 2024-04-16T07:52:56.925 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:52:57.004 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T07:52:57.004 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31598.log 2024-04-16T07:52:57.004 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31220.log 2024-04-16T07:52:57.004 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31598.log: /var/log/ceph/ceph-client.admin.33343.log 2024-04-16T07:52:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31829.log 2024-04-16T07:52:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31894.log 2024-04-16T07:52:57.005 INFO:teuthology.orchestra.run.smithi029.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33343.log.gz 2024-04-16T07:52:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32891.log 2024-04-16T07:52:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31829.log: /var/log/ceph/ceph-client.admin.31894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31432.log 2024-04-16T07:52:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.32891.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31830.log 2024-04-16T07:52:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33028.log 2024-04-16T07:52:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31432.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28990.log 2024-04-16T07:52:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31830.log: /var/log/ceph/ceph-client.admin.33028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31596.log 2024-04-16T07:52:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T07:52:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.28990.log: gzip/var/log/ceph/ceph-client.admin.31596.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31978.log 2024-04-16T07:52:57.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32982.log 2024-04-16T07:52:57.018 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33342.log 2024-04-16T07:52:57.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31978.log: gzip/var/log/ceph/ceph-client.admin.32982.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2024-04-16T07:52:57.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31390.log 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33342.log.gz 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32955.log 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31390.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30587.log 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30815.log 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31292.log 2024-04-16T07:52:57.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30587.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.30815.log: -- /var/log/ceph/ceph-client.admin.30785.log 2024-04-16T07:52:57.020 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31936.log 2024-04-16T07:52:57.020 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31516.log 2024-04-16T07:52:57.020 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30785.log: /var/log/ceph/ceph-client.admin.31936.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T07:52:57.020 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30627.log 2024-04-16T07:52:57.020 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31516.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T07:52:57.020 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33005.log 2024-04-16T07:52:57.021 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.0.log: -- /var/log/ceph/ceph.log 2024-04-16T07:52:57.021 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.33005.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30547.log 2024-04-16T07:52:57.021 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T07:52:57.021 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31724.log 2024-04-16T07:52:57.021 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30547.log: gzip/var/log/ceph/ceph-osd.2.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30349.log 2024-04-16T07:52:57.021 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31597.log 2024-04-16T07:52:57.022 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33093.log 2024-04-16T07:52:57.022 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31723.log 2024-04-16T07:52:57.022 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31684.log/var/log/ceph/ceph-client.admin.33093.log: 2024-04-16T07:52:57.022 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33643.log 2024-04-16T07:52:57.022 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31474.log 2024-04-16T07:52:57.023 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31684.log: /var/log/ceph/ceph-client.admin.33643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30437.log 2024-04-16T07:52:57.023 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33643.log.gz 2024-04-16T07:52:57.023 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31348.log 2024-04-16T07:52:57.023 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30437.log: /var/log/ceph/ceph-client.admin.31348.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31220.log.gz 2024-04-16T07:52:57.023 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31292.log.gz 2024-04-16T07:52:57.023 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-04-16T07:52:57.025 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30587.log.gz 2024-04-16T07:52:57.025 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30627.log.gz 2024-04-16T07:52:57.025 INFO:teuthology.orchestra.run.smithi029.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2024-04-16T07:52:57.025 INFO:teuthology.orchestra.run.smithi029.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33093.log.gz 2024-04-16T07:52:57.026 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30785.log.gz 2024-04-16T07:52:57.026 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30547.log.gz 2024-04-16T07:52:57.029 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32982.log.gz 2024-04-16T07:52:57.029 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31348.log.gz 2024-04-16T07:52:57.042 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31432.log.gz 2024-04-16T07:52:57.043 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31474.log.gz 2024-04-16T07:52:57.043 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31516.log.gz 2024-04-16T07:52:57.043 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31596.log.gz 2024-04-16T07:52:57.043 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31598.log.gz 2024-04-16T07:52:57.043 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31597.log.gz 2024-04-16T07:52:57.044 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31723.log.gz 2024-04-16T07:52:57.044 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31724.log.gz 2024-04-16T07:52:57.044 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31684.log.gz 2024-04-16T07:52:57.044 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31830.log.gz 2024-04-16T07:52:57.044 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31829.log.gz 2024-04-16T07:52:57.044 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31894.log.gz 2024-04-16T07:52:57.045 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31936.log.gz 2024-04-16T07:52:57.045 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31978.log.gz 2024-04-16T07:52:57.045 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30815.log.gz 2024-04-16T07:52:57.045 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T07:52:57.046 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33005.log.gz 2024-04-16T07:52:57.046 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33028.log.gz 2024-04-16T07:52:57.046 INFO:teuthology.orchestra.run.smithi029.stderr: 85.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T07:52:57.049 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32891.log.gz 2024-04-16T07:52:57.050 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32955.log.gz 2024-04-16T07:52:57.070 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30349.log.gz 2024-04-16T07:52:57.070 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30437.log.gz 2024-04-16T07:52:57.127 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T07:52:57.143 INFO:teuthology.orchestra.run.smithi029.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28990.log.gz 2024-04-16T07:52:57.195 INFO:teuthology.orchestra.run.smithi029.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T07:52:59.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-16T07:53:12.175 INFO:teuthology.orchestra.run.smithi029.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T07:53:12.823 INFO:teuthology.orchestra.run.smithi029.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T07:53:12.902 INFO:teuthology.orchestra.run.smithi029.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T07:53:12.904 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-16T07:53:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m15.923s 2024-04-16T07:53:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m5.987s 2024-04-16T07:53:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.352s 2024-04-16T07:53:12.905 INFO:tasks.ceph:Archiving logs... 2024-04-16T07:53:12.905 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001/remote/smithi029/log 2024-04-16T07:53:12.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:53:13.877 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T07:53:13.888 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T07:53:13.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:53:13.953 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T07:53:13.966 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T07:53:13.967 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:53:13.982 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:53:14.053 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:53:14.053 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-16T07:53:14.053 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv01.front.sepia.ceph.com 3 6 377 16 +1477us[+1478us] +/- 43ms 2024-04-16T07:53:14.053 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 -748us[ -748us] +/- 55ms 2024-04-16T07:53:14.053 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -1810us[-1810us] +/- 85ms 2024-04-16T07:53:14.054 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:53:14.079 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T07:53:14.089 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T07:53:14.090 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T07:53:14.099 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:53:14.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:53:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:53:14.315 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T07:53:14.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T07:53:14.333 INFO:teuthology.task.internal:Duration was 1013.298141 seconds 2024-04-16T07:53:14.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T07:53:14.345 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T07:53:14.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:53:14.512 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:53:14.862 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T07:53:14.863 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-04-16T07:53:14.863 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:53:14.884 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T07:53:14.884 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:53:14.955 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T07:53:14.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:53:15.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T07:53:15.107 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T07:53:15.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:53:15.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T07:53:15.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:53:15.224 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-04-16T07:53:15.254 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:53:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:53:15.310 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T07:53:15.320 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T07:53:15.321 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7658001/remote/smithi029 2024-04-16T07:53:15.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T07:53:15.450 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T07:53:15.451 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T07:53:15.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T07:53:15.522 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T07:53:15.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T07:53:15.533 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T07:53:15.533 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:53:15.563 INFO:teuthology.orchestra.run.smithi029.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:53 /home/ubuntu/cephtest 2024-04-16T07:53:15.564 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T07:53:15.574 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T07:53:15.603 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-replicated-april-15&var-job_id_selected=7658001 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} duration: 1013.2981412410736 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-04-16T07:53:15.603 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:53:15.691 INFO:teuthology.run:pass