2024-01-28T13:10:19.296 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-28T13:10:19.297 DEBUG:teuthology.run:Teuthology command: teuthology --name matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi --verbose --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830 --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} -- /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830/orig.config.yaml 2024-01-28T13:10:19.342 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-28T13:10:19.456 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830 branch: wip-matanb-crimson-pg-map-logs description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} email: null first_in_suite: false job_id: '7535830' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-pg-map-logs ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson osd obc lru size: 10 crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 00c0d54d57237e78360214c1fefe250a2610dcfe ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 00c0d54d57237e78360214c1fefe250a2610dcfe workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 7eb9e33f53ab64f23bcf0a2a31509f3442110e4e owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7326 sha1: 00c0d54d57237e78360214c1fefe250a2610dcfe sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_7eb9e33f53ab64f23bcf0a2a31509f3442110e4e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 7eb9e33f53ab64f23bcf0a2a31509f3442110e4e targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSk5Mv474VqVpbJCvzOlBHvdy1h1UC2aszlUiSuKbYnM4+RNzTOSbkTL3ClL7pfs1+8iov/Lmh5srCg+0HgsbE= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randwrite norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-28_12:38:29 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-28T13:10:19.456 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_7eb9e33f53ab64f23bcf0a2a31509f3442110e4e/qa; will attempt to use it 2024-01-28T13:10:19.456 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_7eb9e33f53ab64f23bcf0a2a31509f3442110e4e/qa/tasks 2024-01-28T13:10:19.457 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-28T13:10:19.458 INFO:teuthology.task.internal:Checking packages... 2024-01-28T13:10:19.480 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '00c0d54d57237e78360214c1fefe250a2610dcfe' 2024-01-28T13:10:19.480 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-28T13:10:19.481 INFO:teuthology.packaging:ref: None 2024-01-28T13:10:19.481 INFO:teuthology.packaging:tag: None 2024-01-28T13:10:19.481 INFO:teuthology.packaging:branch: wip-matanb-crimson-pg-map-logs 2024-01-28T13:10:19.481 INFO:teuthology.packaging:sha1: 00c0d54d57237e78360214c1fefe250a2610dcfe 2024-01-28T13:10:19.481 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-pg-map-logs 2024-01-28T13:10:19.620 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-933.g00c0d54d 2024-01-28T13:10:19.622 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-28T13:10:19.628 INFO:teuthology.task.internal:no buildpackages task found 2024-01-28T13:10:19.628 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-28T13:10:19.643 INFO:teuthology.task.internal:Saving configuration 2024-01-28T13:10:19.659 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-28T13:10:19.665 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-28T13:10:19.689 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-28 13:05:27.029699', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSk5Mv474VqVpbJCvzOlBHvdy1h1UC2aszlUiSuKbYnM4+RNzTOSbkTL3ClL7pfs1+8iov/Lmh5srCg+0HgsbE='} 2024-01-28T13:10:19.689 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-28T13:10:19.696 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-28T13:10:19.696 INFO:teuthology.run_tasks:Running task console_log... 2024-01-28T13:10:19.731 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff696450ee0>, signals=[15]) 2024-01-28T13:10:19.732 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-28T13:10:19.738 INFO:teuthology.task.internal:Opening connections... 2024-01-28T13:10:19.738 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:10:19.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-28T13:10:19.843 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-28T13:10:19.849 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-01-28T13:10:19.913 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-01-28T13:10:19.913 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-28T13:10:19.970 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-01-28T13:10:19.971 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-01-28T13:10:19.971 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-28T13:10:19.971 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-01-28T13:10:19.971 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-28T13:10:19.971 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-28T13:10:19.971 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-28T13:10:19.971 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-01-28T13:10:20.001 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-28T13:10:20.010 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-28T13:10:20.015 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-28T13:10:20.015 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-01-28T13:10:20.031 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-28T13:10:20.037 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-28T13:10:20.037 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-01-28T13:10:20.094 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-28T13:10:20.174 INFO:teuthology.run_tasks:Running task kernel... 2024-01-28T13:10:20.188 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-28T13:10:20.188 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-28T13:10:20.188 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-28T13:10:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-28T13:10:20.205 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-01-28T13:10:20.264 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2024-01-28T13:10:20.264 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-361.el9.x86_64 2024-01-28T13:10:20.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-01-28T13:10:22.355 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2024-01-28T13:10:23.730 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-01-28T13:10:25.707 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-01-28T13:10:32.805 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-01-28T13:10:38.041 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.9 MB 00:01 2024-01-28T13:10:39.789 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-01-28T13:10:41.065 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-01-28T13:10:46.199 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-28T13:10:48.589 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-28T13:10:48.589 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-28T13:10:48.693 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-411.el9 CentOS-BaseOS 6.0 M 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-411.el9 CentOS-BaseOS 21 M 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-411.el9 CentOS-BaseOS 39 M 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-411.el9 CentOS-BaseOS 33 M 2024-01-28T13:10:48.708 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 99 M 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 122 M 2024-01-28T13:10:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:10:49.266 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-411.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-01-28T13:10:50.942 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-modules-5.14.0-411.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-01-28T13:10:51.250 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-core-5.14.0-411.el9.x86_6 17 MB/s | 33 MB 00:01 2024-01-28T13:10:51.450 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-core-5.14.0-411.el9.x86_64.rpm 7.6 MB/s | 21 MB 00:02 2024-01-28T13:10:51.453 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:10:51.454 INFO:teuthology.orchestra.run.smithi019.stdout:Total 36 MB/s | 99 MB 00:02 2024-01-28T13:10:51.454 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:10:51.904 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:10:51.904 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:10:52.977 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:10:52.977 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:10:55.519 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:10:56.331 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-411.el9.x86_64 1/4 2024-01-28T13:10:56.477 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-28T13:10:59.325 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-28T13:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-28T13:11:04.560 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-28T13:11:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-411.el9.x86_64 4/4 2024-01-28T13:11:09.901 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-411.el9.x86_64 4/4 2024-01-28T13:11:40.473 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-411.el9.x86_64 4/4 2024-01-28T13:11:40.473 INFO:teuthology.orchestra.run.smithi019.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-28T13:11:40.473 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:11:40.543 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-411.el9.x86_64 4/4 2024-01-28T13:11:41.003 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-411.el9.x86_64 4/4 2024-01-28T13:11:41.004 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-411.el9.x86_64 1/4 2024-01-28T13:11:41.004 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-28T13:11:41.004 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-411.el9.x86_64 4/4 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-411.el9.x86_64 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-411.el9.x86_64 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-411.el9.x86_64 2024-01-28T13:11:41.455 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-411.el9.x86_64 2024-01-28T13:11:41.456 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:11:41.456 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:11:41.964 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-01-28T13:11:42.100 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-411.el9.x86_64 2024-01-28T13:11:42.101 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-411.el9.x86_64 2024-01-28T13:11:42.101 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-411.el9.x86_64 2024-01-28T13:11:42.101 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-28T13:11:42.101 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-28T13:11:42.101 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-28T13:11:42.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2024-01-28T13:11:42.101 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-01-28T13:11:42.215 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-411.el9.x86_64 2024-01-28T13:11:42.215 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-411.el9.x86_64 2024-01-28T13:11:42.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-01-28T13:11:42.253 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-28T13:11:42.254 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-01-28T13:11:42.255 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:11:42.255 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-01-28T13:11:42.255 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-28T13:11:42.256 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-411.el9.x86_64 2024-01-28T13:11:42.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-28T13:11:42.973 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-01-28T13:11:44.429 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-28T13:11:44.468 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-01-28T13:11:44.471 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-01-28T13:11:44.486 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.gsA7vavMeX 2024-01-28T13:11:44.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gsA7vavMeX 2024-01-28T13:11:44.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.gsA7vavMeX 2024-01-28T13:11:44.705 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.gsA7vavMeX is 6KB 2024-01-28T13:11:44.753 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.gsA7vavMeX 2024-01-28T13:11:44.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-01-28T13:11:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-28T13:11:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-28T13:11:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-28T13:11:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-411.el9.x86_64.conf 2024-01-28T13:11:44.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-411.el9.x86_64 2024-01-28T13:11:44.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-01-28T13:12:15.013 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-28T13:12:15.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-28T13:12:15.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-28T13:12:33.340 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2024-01-28T13:12:42.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-28T13:12:42.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-28T13:13:42.401 DEBUG:teuthology.orchestra.remote:timed out 2024-01-28T13:13:54.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-28T13:13:54.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-28T13:13:54.772 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-01-28T13:13:55.154 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-28T13:13:55.154 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-411.el9.x86_64"... 2024-01-28T13:13:55.155 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-01-28T13:13:55.170 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-411.el9.x86_64 2024-01-28T13:13:55.170 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-411.el9.x86_64 vs 5.14.0-411.el9.x86_64 2024-01-28T13:13:55.170 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-28T13:13:55.170 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-28T13:13:56.171 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-28T13:13:56.171 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-28T13:13:56.229 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-01-28T13:13:56.265 DEBUG:teuthology.parallel:result is None 2024-01-28T13:13:56.265 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-28T13:13:56.274 INFO:teuthology.task.internal:Creating test directory... 2024-01-28T13:13:56.274 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-28T13:13:56.291 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-28T13:13:56.298 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-28T13:13:56.312 INFO:teuthology.task.internal:Creating archive directory... 2024-01-28T13:13:56.312 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-28T13:13:56.359 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-28T13:13:56.371 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-28T13:13:56.371 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-28T13:13:56.433 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-28T13:13:56.447 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-28T13:13:56.448 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-28T13:13:56.466 INFO:teuthology.task.internal:Configuring sudo... 2024-01-28T13:13:56.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-28T13:13:56.517 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-28T13:13:56.535 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-28T13:13:56.536 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-28T13:13:56.572 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-28T13:13:56.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-28T13:13:56.704 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-28T13:13:56.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-28T13:13:56.829 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:13:56.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-28T13:13:56.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-01-28T13:13:56.982 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-28T13:13:57.165 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-28T13:13:57.174 INFO:teuthology.task.internal:Starting timer... 2024-01-28T13:13:57.174 INFO:teuthology.run_tasks:Running task pcp... 2024-01-28T13:13:57.197 INFO:teuthology.run_tasks:Running task selinux... 2024-01-28T13:13:57.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-01-28T13:13:57.260 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-01-28T13:13:57.261 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-28T13:13:57.261 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-01-28T13:13:57.299 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-01-28T13:13:57.299 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive'} 2024-01-28T13:13:57.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-28T13:13:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-28T13:13:57.368 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-28T13:13:57.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-01-28T13:13:57.451 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-28T13:13:57.461 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-28T13:13:57.475 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-28T13:13:57.476 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-28T13:18:05.154 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com')] 2024-01-28T13:18:05.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-28T13:18:05.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-28T13:18:05.249 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-01-28T13:18:05.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-28T13:18:05.323 INFO:teuthology.run_tasks:Running task clock... 2024-01-28T13:18:05.334 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-28T13:18:05.334 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-28T13:18:05.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-28T13:18:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-28T13:18:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-28T13:18:05.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-01-28T13:18:05.452 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-01-28T13:18:05.469 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-28T13:18:05.486 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-28T13:18:05.549 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-01-28T13:18:05.552 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-28T13:18:05.552 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-01-28T13:18:05.552 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-28T13:18:05.552 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-28T13:18:05.552 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-28T13:18:05.553 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-28T13:18:05.553 INFO:teuthology.run_tasks:Running task install... 2024-01-28T13:18:05.561 DEBUG:teuthology.task.install:project ceph 2024-01-28T13:18:05.561 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe'}} 2024-01-28T13:18:05.562 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe'} 2024-01-28T13:18:05.562 INFO:teuthology.task.install:Using flavor: crimson 2024-01-28T13:18:05.567 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-28T13:18:05.568 INFO:teuthology.task.install:extra packages: [] 2024-01-28T13:18:05.568 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe', 'tag': None, 'wait_for_package': False} 2024-01-28T13:18:05.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=00c0d54d57237e78360214c1fefe250a2610dcfe 2024-01-28T13:18:05.714 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-pg-map-logs/00c0d54d57237e78360214c1fefe250a2610dcfe/centos/9/flavors/crimson/ 2024-01-28T13:18:05.715 INFO:teuthology.task.install.rpm:Package version is 19.0.0-933.g00c0d54d 2024-01-28T13:18:05.824 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-pg-map-logs/00c0d54d57237e78360214c1fefe250a2610dcfe/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-pg-map-logs/00c0d54d57237e78360214c1fefe250a2610dcfe/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-pg-map-logs/00c0d54d57237e78360214c1fefe250a2610dcfe/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-28T13:18:05.825 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:18:05.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-28T13:18:05.852 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-28T13:18:05.853 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/00c0d54d57237e78360214c1fefe250a2610dcfe/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-28T13:18:05.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-28T13:18:06.001 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-28T13:18:06.066 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-01-28T13:18:06.068 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-01-28T13:18:06.319 INFO:teuthology.orchestra.run.smithi019.stdout:74 files removed 2024-01-28T13:18:06.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-01-28T13:18:07.078 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 204 kB/s | 83 kB 00:00 2024-01-28T13:18:07.421 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 33 kB/s | 9.5 kB 00:00 2024-01-28T13:18:07.805 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-01-28T13:18:08.224 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2024-01-28T13:18:09.724 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-01-28T13:18:11.557 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-01-28T13:18:14.917 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-01-28T13:18:19.733 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 5.9 MB 00:01 2024-01-28T13:18:21.134 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 15 kB 00:00 2024-01-28T13:18:22.061 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-01-28T13:18:27.203 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-01-28T13:18:30.025 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:18:30.033 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 9.0 M 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 3.1 M 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 3.1 M 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 5.1 M 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 18 M 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 25 k 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-28T13:18:30.034 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 752 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 497 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 4.8 M 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 45 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 130 k 2024-01-28T13:18:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 162 k 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 321 k 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 298 k 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 99 k 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout:Install 27 Packages 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-01-28T13:18:30.036 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:30.037 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 54 M 2024-01-28T13:18:30.037 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:18:30.779 INFO:teuthology.orchestra.run.smithi019.stdout:(1/29): ceph-base-19.0.0-933.g00c0d54d.el9.x86_ 7.8 MB/s | 5.1 MB 00:00 2024-01-28T13:18:30.838 INFO:teuthology.orchestra.run.smithi019.stdout:(2/29): ceph-selinux-19.0.0-933.g00c0d54d.el9.x 432 kB/s | 25 kB 00:00 2024-01-28T13:18:30.946 INFO:teuthology.orchestra.run.smithi019.stdout:(3/29): libcephfs2-19.0.0-933.g00c0d54d.el9.x86 6.8 MB/s | 752 kB 00:00 2024-01-28T13:18:31.063 INFO:teuthology.orchestra.run.smithi019.stdout:(4/29): ceph-radosgw-19.0.0-933.g00c0d54d.el9.x 9.6 MB/s | 9.0 MB 00:00 2024-01-28T13:18:31.097 INFO:teuthology.orchestra.run.smithi019.stdout:(5/29): libradosstriper1-19.0.0-933.g00c0d54d.e 3.2 MB/s | 497 kB 00:00 2024-01-28T13:18:31.164 INFO:teuthology.orchestra.run.smithi019.stdout:(6/29): python3-ceph-argparse-19.0.0-933.g00c0d 675 kB/s | 45 kB 00:00 2024-01-28T13:18:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:(7/29): python3-ceph-common-19.0.0-933.g00c0d54 1.5 MB/s | 130 kB 00:00 2024-01-28T13:18:31.364 INFO:teuthology.orchestra.run.smithi019.stdout:(8/29): librgw2-19.0.0-933.g00c0d54d.el9.x86_64 16 MB/s | 4.8 MB 00:00 2024-01-28T13:18:31.389 INFO:teuthology.orchestra.run.smithi019.stdout:(9/29): python3-cephfs-19.0.0-933.g00c0d54d.el9 1.1 MB/s | 162 kB 00:00 2024-01-28T13:18:31.415 INFO:teuthology.orchestra.run.smithi019.stdout:(10/29): python3-rados-19.0.0-933.g00c0d54d.el9 6.2 MB/s | 321 kB 00:00 2024-01-28T13:18:31.440 INFO:teuthology.orchestra.run.smithi019.stdout:(11/29): python3-rbd-19.0.0-933.g00c0d54d.el9.x 5.8 MB/s | 298 kB 00:00 2024-01-28T13:18:31.465 INFO:teuthology.orchestra.run.smithi019.stdout:(12/29): python3-rgw-19.0.0-933.g00c0d54d.el9.x 1.9 MB/s | 99 kB 00:00 2024-01-28T13:18:31.591 INFO:teuthology.orchestra.run.smithi019.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-01-28T13:18:31.616 INFO:teuthology.orchestra.run.smithi019.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-01-28T13:18:31.633 INFO:teuthology.orchestra.run.smithi019.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-01-28T13:18:31.725 INFO:teuthology.orchestra.run.smithi019.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 661 kB 00:00 2024-01-28T13:18:31.750 INFO:teuthology.orchestra.run.smithi019.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-01-28T13:18:31.776 INFO:teuthology.orchestra.run.smithi019.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 822 kB/s | 41 kB 00:00 2024-01-28T13:18:31.976 INFO:teuthology.orchestra.run.smithi019.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-01-28T13:18:32.035 INFO:teuthology.orchestra.run.smithi019.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-01-28T13:18:32.085 INFO:teuthology.orchestra.run.smithi019.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-01-28T13:18:32.135 INFO:teuthology.orchestra.run.smithi019.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-01-28T13:18:32.294 INFO:teuthology.orchestra.run.smithi019.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-01-28T13:18:32.344 INFO:teuthology.orchestra.run.smithi019.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-01-28T13:18:32.695 INFO:teuthology.orchestra.run.smithi019.stdout:(25/29): ceph-common-19.0.0-933.g00c0d54d.el9.x 7.0 MB/s | 18 MB 00:02 2024-01-28T13:18:32.736 INFO:teuthology.orchestra.run.smithi019.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-01-28T13:18:32.887 INFO:teuthology.orchestra.run.smithi019.stdout:(27/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-01-28T13:18:33.070 INFO:teuthology.orchestra.run.smithi019.stdout:(28/29): librados2-19.0.0-933.g00c0d54d.el9.x86 8.4 MB/s | 3.1 MB 00:00 2024-01-28T13:18:33.120 INFO:teuthology.orchestra.run.smithi019.stdout:(29/29): librbd1-19.0.0-933.g00c0d54d.el9.x86_6 8.0 MB/s | 3.1 MB 00:00 2024-01-28T13:18:33.126 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:18:33.126 INFO:teuthology.orchestra.run.smithi019.stdout:Total 18 MB/s | 54 MB 00:03 2024-01-28T13:18:33.127 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:18:33.162 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:18:33.163 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:18:33.643 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:18:33.643 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:18:35.199 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:18:35.624 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-28T13:18:35.673 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.0.0-933.g00c0d54d.el9.x86_64 2/31 2024-01-28T13:18:35.757 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.0.0-933.g00c0d54d.el9.x86_64 2/31 2024-01-28T13:18:35.858 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.0.0-933.g00c0d54d.el9.x86_64 3/31 2024-01-28T13:18:35.882 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.0.0-933.g00c0d54d.el9.x86_64 4/31 2024-01-28T13:18:35.965 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.0.0-933.g00c0d54d.el9.x86_64 4/31 2024-01-28T13:18:35.995 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-28T13:18:36.020 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-28T13:18:36.058 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-28T13:18:36.102 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.0.0-933.g00c0d54d.el9.x 8/31 2024-01-28T13:18:36.174 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.0.0-933.g00c0d54d.el9.x86_64 9/31 2024-01-28T13:18:36.203 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.0.0-933.g00c0d54d.el9.x86_64 10/31 2024-01-28T13:18:36.274 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.0.0-933.g00c0d54d.el9.x86_64 10/31 2024-01-28T13:18:36.310 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-28T13:18:36.421 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-28T13:18:36.461 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-28T13:18:36.502 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-28T13:18:36.550 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-28T13:18:36.579 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-28T13:18:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-28T13:18:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-28T13:18:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.0.0-933.g00c0d54d.el9.x86_64 19/31 2024-01-28T13:18:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.0.0-933.g00c0d54d.el9.x86_64 19/31 2024-01-28T13:18:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.0.0-933.g00c0d54d.el9.x86_64 20/31 2024-01-28T13:18:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-28T13:18:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-28T13:18:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.0.0-933.g00c0d54d.el9.x86_64 23/31 2024-01-28T13:18:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.0.0-933.g00c0d54d.el9.x86_64 23/31 2024-01-28T13:18:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.0.0-933.g00c0d54d.el9.x86_64 24/31 2024-01-28T13:18:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.0.0-933.g00c0d54d.el9.x86 25/31 2024-01-28T13:18:40.076 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-933.g00c0d54d.el9.x86_64 26/31 2024-01-28T13:18:40.111 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.0.0-933.g00c0d54d.el9.x86_64 26/31 2024-01-28T13:18:40.682 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-933.g00c0d54d.el9.x86_64 26/31 2024-01-28T13:18:40.722 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.0.0-933.g00c0d54d.el9.x86_64 27/31 2024-01-28T13:18:40.813 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.0.0-933.g00c0d54d.el9.x86_64 27/31 2024-01-28T13:18:40.813 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-28T13:18:40.813 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-28T13:18:40.813 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:40.840 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.0.0-933.g00c0d54d.el9.x86_64 28/31 2024-01-28T13:18:50.967 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.0.0-933.g00c0d54d.el9.x86_64 28/31 2024-01-28T13:18:50.967 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-01-28T13:18:50.968 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-01-28T13:18:50.968 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-01-28T13:18:50.968 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-01-28T13:18:50.968 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-01-28T13:18:50.968 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-01-28T13:18:50.968 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-01-28T13:18:50.969 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:51.889 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.0.0-933.g00c0d54d.el9.x86_64 29/31 2024-01-28T13:18:51.910 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-933.g00c0d54d.el9.x86_64 29/31 2024-01-28T13:18:51.910 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:18:51.910 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-28T13:18:51.910 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-28T13:18:51.910 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-28T13:18:51.910 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:51.912 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-28T13:18:52.000 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-28T13:18:52.000 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-28T13:18:53.263 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.0.0-933.g00c0d54d.el9.x86_64 1/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.0.0-933.g00c0d54d.el9.x86_64 2/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.0.0-933.g00c0d54d.el9.x86_64 3/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.0.0-933.g00c0d54d.el9.x86_64 4/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.0.0-933.g00c0d54d.el9.x86_64 5/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.0.0-933.g00c0d54d.el9.x86_64 6/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.0.0-933.g00c0d54d.el9.x86_64 7/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.0.0-933.g00c0d54d.el9.x 8/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.0.0-933.g00c0d54d.el9.x86 9/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.0.0-933.g00c0d54d.el9.x86_64 10/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.0.0-933.g00c0d54d.el9.x86_64 11/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.0.0-933.g00c0d54d.el9.x86_64 12/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.0.0-933.g00c0d54d.el9.x86_64 13/31 2024-01-28T13:18:53.264 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.0.0-933.g00c0d54d.el9.x86_64 28/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-28T13:18:53.265 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.0.0-933.g00c0d54d.el9.x86_64 30/31 2024-01-28T13:18:54.159 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-28T13:18:54.159 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:54.159 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-28T13:18:54.160 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:18:54.162 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2024-01-28T13:18:54.162 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-28T13:18:54.162 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:54.162 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:18:54.398 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-01-28T13:18:54.981 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:27 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:18:55.141 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:18:55.141 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 121 M 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 158 k 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-28T13:18:55.142 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout:Install 10 Packages 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:18:55.143 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 123 M 2024-01-28T13:18:55.144 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.1 G 2024-01-28T13:18:55.144 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:18:55.389 INFO:teuthology.orchestra.run.smithi019.stdout:(1/10): libasan-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 417 kB 00:00 2024-01-28T13:18:55.439 INFO:teuthology.orchestra.run.smithi019.stdout:(2/10): libubsan-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-01-28T13:18:55.498 INFO:teuthology.orchestra.run.smithi019.stdout:(3/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-01-28T13:18:55.523 INFO:teuthology.orchestra.run.smithi019.stdout:(4/10): libcephsqlite-19.0.0-933.g00c0d54d.el9. 484 kB/s | 158 kB 00:00 2024-01-28T13:18:55.548 INFO:teuthology.orchestra.run.smithi019.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-01-28T13:18:55.573 INFO:teuthology.orchestra.run.smithi019.stdout:(6/10): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-01-28T13:18:55.599 INFO:teuthology.orchestra.run.smithi019.stdout:(7/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-01-28T13:18:55.816 INFO:teuthology.orchestra.run.smithi019.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 562 kB/s | 122 kB 00:00 2024-01-28T13:18:56.050 INFO:teuthology.orchestra.run.smithi019.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.6 MB/s | 1.3 MB 00:00 2024-01-28T13:19:22.148 INFO:teuthology.orchestra.run.smithi019.stdout:(10/10): ceph-test-19.0.0-933.g00c0d54d.el9.x86 4.5 MB/s | 121 MB 00:26 2024-01-28T13:19:22.150 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:19:22.151 INFO:teuthology.orchestra.run.smithi019.stdout:Total 4.6 MB/s | 123 MB 00:27 2024-01-28T13:19:22.151 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:19:22.174 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:19:22.175 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:19:23.040 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:19:23.040 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:19:24.213 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:19:24.288 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-28T13:19:24.333 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-28T13:19:24.373 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/10 2024-01-28T13:19:24.406 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-16.el9.x86_64 4/10 2024-01-28T13:19:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2024-01-28T13:19:24.479 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/10 2024-01-28T13:19:24.509 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 7/10 2024-01-28T13:19:24.550 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libasan-11.4.1-3.el9.x86_64 8/10 2024-01-28T13:19:24.577 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.0.0-933.g00c0d54d.el9.x86_64 9/10 2024-01-28T13:19:42.257 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.0.0-933.g00c0d54d.el9.x86_64 9/10 2024-01-28T13:19:42.328 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.0.0-933.g00c0d54d.el9.x86_64 10/10 2024-01-28T13:19:50.453 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:19.0.0-933.g00c0d54d.el9.x86_64 10/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.0.0-933.g00c0d54d.el9.x86_64 1/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.0.0-933.g00c0d54d.el9.x86_64 2/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-16.el9.x86_64 7/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/10 2024-01-28T13:19:50.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.0.0-933.g00c0d54d.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.0.0-933.g00c0d54d.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-28T13:19:50.948 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:19:50.949 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:19:51.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-01-28T13:19:51.861 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:24 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:19:52.027 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:19:52.029 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 6.5 k 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 2.0 M 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 1.5 M 2024-01-28T13:19:52.030 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 245 k 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 4.8 M 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 117 M 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-28T13:19:52.031 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-28T13:19:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-28T13:19:52.033 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-28T13:19:52.034 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-28T13:19:52.035 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:19:52.035 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:19:52.035 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-01-28T13:19:52.035 INFO:teuthology.orchestra.run.smithi019.stdout:Install 36 Packages 2024-01-28T13:19:52.035 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:19:52.036 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 130 M 2024-01-28T13:19:52.036 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.4 G 2024-01-28T13:19:52.036 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:19:52.694 INFO:teuthology.orchestra.run.smithi019.stdout:(1/36): ceph-19.0.0-933.g00c0d54d.el9.x86_64.rp 39 kB/s | 6.5 kB 00:00 2024-01-28T13:19:52.877 INFO:teuthology.orchestra.run.smithi019.stdout:(2/36): ceph-mgr-19.0.0-933.g00c0d54d.el9.x86_6 4.2 MB/s | 1.5 MB 00:00 2024-01-28T13:19:53.161 INFO:teuthology.orchestra.run.smithi019.stdout:(3/36): ceph-mds-19.0.0-933.g00c0d54d.el9.x86_6 3.2 MB/s | 2.0 MB 00:00 2024-01-28T13:19:53.270 INFO:teuthology.orchestra.run.smithi019.stdout:(4/36): ceph-mgr-modules-core-19.0.0-933.g00c0d 2.2 MB/s | 245 kB 00:00 2024-01-28T13:19:53.403 INFO:teuthology.orchestra.run.smithi019.stdout:(5/36): ceph-mon-19.0.0-933.g00c0d54d.el9.x86_6 6.8 MB/s | 4.8 MB 00:00 2024-01-28T13:19:53.428 INFO:teuthology.orchestra.run.smithi019.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-01-28T13:19:53.454 INFO:teuthology.orchestra.run.smithi019.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-01-28T13:19:53.479 INFO:teuthology.orchestra.run.smithi019.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 627 kB/s | 32 kB 00:00 2024-01-28T13:19:53.505 INFO:teuthology.orchestra.run.smithi019.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 939 kB/s | 47 kB 00:00 2024-01-28T13:19:53.530 INFO:teuthology.orchestra.run.smithi019.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-01-28T13:19:53.555 INFO:teuthology.orchestra.run.smithi019.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-28T13:19:53.597 INFO:teuthology.orchestra.run.smithi019.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-01-28T13:19:53.622 INFO:teuthology.orchestra.run.smithi019.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-01-28T13:19:53.647 INFO:teuthology.orchestra.run.smithi019.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-28T13:19:53.673 INFO:teuthology.orchestra.run.smithi019.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-28T13:19:53.701 INFO:teuthology.orchestra.run.smithi019.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.4 MB/s | 187 kB 00:00 2024-01-28T13:19:53.769 INFO:teuthology.orchestra.run.smithi019.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 235 kB/s | 22 kB 00:00 2024-01-28T13:19:53.894 INFO:teuthology.orchestra.run.smithi019.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 787 kB/s | 151 kB 00:00 2024-01-28T13:19:53.910 INFO:teuthology.orchestra.run.smithi019.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 307 kB/s | 43 kB 00:00 2024-01-28T13:19:53.969 INFO:teuthology.orchestra.run.smithi019.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-01-28T13:19:54.011 INFO:teuthology.orchestra.run.smithi019.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-01-28T13:19:54.061 INFO:teuthology.orchestra.run.smithi019.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-01-28T13:19:54.095 INFO:teuthology.orchestra.run.smithi019.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-01-28T13:19:54.120 INFO:teuthology.orchestra.run.smithi019.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-01-28T13:19:54.146 INFO:teuthology.orchestra.run.smithi019.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-01-28T13:19:54.171 INFO:teuthology.orchestra.run.smithi019.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-01-28T13:19:54.196 INFO:teuthology.orchestra.run.smithi019.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-01-28T13:19:54.221 INFO:teuthology.orchestra.run.smithi019.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-01-28T13:19:54.263 INFO:teuthology.orchestra.run.smithi019.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-01-28T13:19:54.288 INFO:teuthology.orchestra.run.smithi019.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-01-28T13:19:54.331 INFO:teuthology.orchestra.run.smithi019.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-01-28T13:19:54.347 INFO:teuthology.orchestra.run.smithi019.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-01-28T13:19:54.389 INFO:teuthology.orchestra.run.smithi019.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-01-28T13:19:54.423 INFO:teuthology.orchestra.run.smithi019.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-01-28T13:19:54.448 INFO:teuthology.orchestra.run.smithi019.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-01-28T13:19:58.307 INFO:teuthology.orchestra.run.smithi019.stdout:(36/36): ceph-osd-19.0.0-933.g00c0d54d.el9.x86_ 22 MB/s | 117 MB 00:05 2024-01-28T13:19:58.311 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:19:58.311 INFO:teuthology.orchestra.run.smithi019.stdout:Total 21 MB/s | 130 MB 00:06 2024-01-28T13:19:58.333 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:19:58.380 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:19:58.380 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:19:59.426 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:19:59.426 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:20:01.360 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:20:01.418 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-28T13:20:01.451 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-28T13:20:01.478 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-28T13:20:01.550 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-28T13:20:01.594 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-28T13:20:01.627 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-28T13:20:01.721 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-28T13:20:01.806 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-28T13:20:01.846 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-28T13:20:01.919 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-28T13:20:01.972 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-28T13:20:02.041 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-28T13:20:02.093 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-28T13:20:02.148 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-28T13:20:02.338 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-28T13:20:02.381 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-28T13:20:02.419 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-28T13:20:02.453 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-28T13:20:02.483 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-28T13:20:02.562 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-28T13:20:02.594 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-28T13:20:02.627 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-28T13:20:02.668 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-28T13:20:02.703 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-28T13:20:02.737 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-28T13:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-28T13:20:02.868 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-28T13:20:02.942 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-28T13:20:03.081 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-28T13:20:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-28T13:20:03.377 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-933.g00c0d54d.el9.n 31/36 2024-01-28T13:20:03.414 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.0.0-933.g00c0d54d.el9.x86_64 32/36 2024-01-28T13:20:03.455 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.0.0-933.g00c0d54d.el9.x86_64 32/36 2024-01-28T13:20:03.455 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:20:03.455 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-28T13:20:03.456 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-28T13:20:03.456 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-28T13:20:03.456 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:23.022 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.0.0-933.g00c0d54d.el9.x86_64 33/36 2024-01-28T13:20:23.048 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.0.0-933.g00c0d54d.el9.x86_64 33/36 2024-01-28T13:20:23.048 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:20:23.048 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-28T13:20:23.048 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-28T13:20:23.048 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-28T13:20:23.048 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:23.531 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.0.0-933.g00c0d54d.el9.x86_64 34/36 2024-01-28T13:20:23.553 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.0.0-933.g00c0d54d.el9.x86_64 34/36 2024-01-28T13:20:23.553 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:20:23.553 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-28T13:20:23.553 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-28T13:20:23.553 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-28T13:20:23.554 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.0.0-933.g00c0d54d.el9.x86_64 35/36 2024-01-28T13:20:23.788 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.0.0-933.g00c0d54d.el9.x86_64 35/36 2024-01-28T13:20:23.788 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:20:23.788 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-28T13:20:23.788 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-28T13:20:23.788 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-28T13:20:23.789 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:23.857 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.0.0-933.g00c0d54d.el9.x86_64 36/36 2024-01-28T13:20:33.385 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:19.0.0-933.g00c0d54d.el9.x86_64 36/36 2024-01-28T13:20:33.385 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.0.0-933.g00c0d54d.el9.x86_64 1/36 2024-01-28T13:20:33.385 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.0.0-933.g00c0d54d.el9.x86_64 2/36 2024-01-28T13:20:33.385 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.0.0-933.g00c0d54d.el9.x86_64 3/36 2024-01-28T13:20:33.385 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.0.0-933.g00c0d54d.el9.x86_64 4/36 2024-01-28T13:20:33.385 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.0.0-933.g00c0d54d.el9.x86_64 5/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-933.g00c0d54d.el9.n 6/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-28T13:20:33.386 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-28T13:20:33.387 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-28T13:20:33.388 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-28T13:20:33.388 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-28T13:20:33.388 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:20:34.512 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-28T13:20:34.513 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-28T13:20:34.514 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:34.515 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:34.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-01-28T13:20:35.342 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:08 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:35.447 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:20:35.500 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:35.502 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:20:35.502 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:35.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-01-28T13:20:36.147 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:09 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:36.308 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 743 k 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:36.309 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:20:36.310 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:36.310 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:20:36.310 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:36.310 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 743 k 2024-01-28T13:20:36.310 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 748 k 2024-01-28T13:20:36.310 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:20:36.747 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-19.0.0-933.g00c0d54d.el9.noarch.rpm 1.7 MB/s | 743 kB 00:00 2024-01-28T13:20:36.748 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:20:36.748 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.7 MB/s | 743 kB 00:00 2024-01-28T13:20:36.748 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:20:36.753 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:20:36.754 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:20:36.759 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:20:36.760 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:20:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:20:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 1/1 2024-01-28T13:20:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 1/1 2024-01-28T13:20:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 1/1 2024-01-28T13:20:38.149 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 1/1 2024-01-28T13:20:38.149 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:38.149 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:20:38.149 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:20:38.149 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:38.150 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:38.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-01-28T13:20:38.910 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:11 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:39.067 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:39.067 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:39.067 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 147 k 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 147 k 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 481 k 2024-01-28T13:20:39.068 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:20:39.286 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-19.0.0-933.g00c0d54 677 kB/s | 147 kB 00:00 2024-01-28T13:20:39.287 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:20:39.287 INFO:teuthology.orchestra.run.smithi019.stdout:Total 673 kB/s | 147 kB 00:00 2024-01-28T13:20:39.287 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:20:39.294 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:20:39.294 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:20:39.348 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:20:39.348 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:20:39.513 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:20:39.562 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-933.g00c0d54d.e 1/1 2024-01-28T13:20:39.586 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-933.g00c0d54d.e 1/1 2024-01-28T13:20:39.586 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:20:39.586 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-28T13:20:39.586 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:40.588 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-933.g00c0d54d.e 1/1 2024-01-28T13:20:40.589 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:40.589 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:20:40.589 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:20:40.589 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:40.589 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:40.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-01-28T13:20:41.377 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:14 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:41.481 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:20:41.535 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:41.537 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:20:41.537 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:41.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-01-28T13:20:42.182 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:15 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:42.341 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 1.7 M 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 28 k 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 15 k 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:42.342 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:20:42.343 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:20:42.343 INFO:teuthology.orchestra.run.smithi019.stdout:Install 5 Packages 2024-01-28T13:20:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:42.343 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2024-01-28T13:20:42.343 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 8.5 M 2024-01-28T13:20:42.343 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:20:42.748 INFO:teuthology.orchestra.run.smithi019.stdout:(1/5): ceph-grafana-dashboards-19.0.0-933.g00c0 110 kB/s | 28 kB 00:00 2024-01-28T13:20:42.773 INFO:teuthology.orchestra.run.smithi019.stdout:(2/5): ceph-prometheus-alerts-19.0.0-933.g00c0d 53 kB/s | 15 kB 00:00 2024-01-28T13:20:42.848 INFO:teuthology.orchestra.run.smithi019.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 307 kB/s | 31 kB 00:00 2024-01-28T13:20:42.915 INFO:teuthology.orchestra.run.smithi019.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-01-28T13:20:43.082 INFO:teuthology.orchestra.run.smithi019.stdout:(5/5): ceph-mgr-dashboard-19.0.0-933.g00c0d54d. 2.9 MB/s | 1.7 MB 00:00 2024-01-28T13:20:43.083 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:20:43.084 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-01-28T13:20:43.084 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:20:43.091 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:20:43.091 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:20:43.156 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:20:43.156 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:20:43.391 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:20:43.464 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-28T13:20:43.495 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-28T13:20:43.525 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-933.g00c0d54d.el9.no 3/5 2024-01-28T13:20:43.772 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-933.g00c0d54d.el9.n 4/5 2024-01-28T13:20:43.801 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-933.g00c0d54d.el9.noarch 5/5 2024-01-28T13:20:44.345 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-933.g00c0d54d.el9.noarch 5/5 2024-01-28T13:20:44.346 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-933.g00c0d54d.el9.n 1/5 2024-01-28T13:20:44.346 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-933.g00c0d54d.el9.noarch 2/5 2024-01-28T13:20:44.346 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-933.g00c0d54d.el9.no 3/5 2024-01-28T13:20:44.346 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:20:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-28T13:20:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-28T13:20:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:44.641 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:44.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-28T13:20:45.427 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:18 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:45.587 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:45.588 INFO:teuthology.orchestra.run.smithi019.stdout:=========================================================================================== 2024-01-28T13:20:45.588 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:20:45.588 INFO:teuthology.orchestra.run.smithi019.stdout:=========================================================================================== 2024-01-28T13:20:45.588 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:20:45.588 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 7.4 M 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-28T13:20:45.589 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout:=========================================================================================== 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 37 M 2024-01-28T13:20:45.590 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 209 M 2024-01-28T13:20:45.591 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:20:45.847 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-01-28T13:20:45.881 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2024-01-28T13:20:45.922 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 847 kB/s | 34 kB 00:00 2024-01-28T13:20:45.972 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-28T13:20:46.165 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-01-28T13:20:46.240 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-01-28T13:20:46.273 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 6.4 MB/s | 217 kB 00:00 2024-01-28T13:20:46.424 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.3 MB/s | 802 kB 00:00 2024-01-28T13:20:46.574 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-01-28T13:20:46.599 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 194 kB 00:00 2024-01-28T13:20:46.941 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-9 6.0 MB/s | 7.4 MB 00:01 2024-01-28T13:20:47.433 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 286 kB/s | 245 kB 00:00 2024-01-28T13:20:47.435 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:20:47.436 INFO:teuthology.orchestra.run.smithi019.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-28T13:20:47.449 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:20:47.476 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:20:47.477 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:20:47.860 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:20:47.861 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:20:48.585 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:20:48.670 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-28T13:20:48.712 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-28T13:20:48.810 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-28T13:20:48.849 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-28T13:20:49.035 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-28T13:20:49.073 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-28T13:20:49.173 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-28T13:20:49.885 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-28T13:20:49.996 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-28T13:20:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-28T13:20:52.392 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-28T13:20:52.422 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-933.g00c0d5 12/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-933.g00c0d5 12/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-933.g00c0d5 1/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-28T13:20:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-28T13:20:54.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-28T13:20:54.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-28T13:20:54.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-28T13:20:54.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-28T13:20:54.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-28T13:20:55.406 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-28T13:20:55.406 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:55.406 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:20:55.406 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:55.407 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:20:55.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-01-28T13:20:56.392 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:29 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:20:56.555 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:20:56.556 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-01-28T13:20:56.556 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:20:56.556 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-01-28T13:20:56.556 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:20:56.556 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 52 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:56.557 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:20:56.558 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-01-28T13:20:56.558 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-01-28T13:20:56.558 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:20:56.558 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2024-01-28T13:20:56.558 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-01-28T13:20:56.558 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:20:56.747 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2024-01-28T13:20:56.772 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 24 kB 00:00 2024-01-28T13:20:56.814 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-01-28T13:20:56.840 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-01-28T13:20:56.865 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-28T13:20:56.890 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-01-28T13:20:56.915 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): ceph-mgr-rook-19.0.0-933.g00c0d54d.el9. 171 kB/s | 52 kB 00:00 2024-01-28T13:20:56.974 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-01-28T13:20:56.999 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-01-28T13:20:57.058 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-01-28T13:20:57.083 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-google-auth-2.26.2-1.el9.noarc 1.2 MB/s | 202 kB 00:00 2024-01-28T13:20:57.108 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-01-28T13:20:57.142 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-01-28T13:20:57.144 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:20:57.144 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2024-01-28T13:20:57.144 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:20:57.158 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:20:57.158 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:20:57.260 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:20:57.261 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:20:57.782 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:20:57.963 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-28T13:20:58.028 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-28T13:20:58.084 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-28T13:20:58.120 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-28T13:20:58.162 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-28T13:20:58.258 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-28T13:20:58.370 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 7/13 2024-01-28T13:20:58.422 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-28T13:20:59.179 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-28T13:20:59.226 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-28T13:20:59.260 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-28T13:20:59.309 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-28T13:20:59.338 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.0.0-933.g00c0d54d.el9.noarch 13/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-933.g00c0d54d.el9.noarch 13/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.0.0-933.g00c0d54d.el9.noarch 1/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-28T13:21:00.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-28T13:21:00.111 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-28T13:21:00.111 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-28T13:21:00.111 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-28T13:21:00.111 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 10/13 2024-01-28T13:21:00.111 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-28T13:21:00.111 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-28T13:21:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-28T13:21:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:00.480 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-28T13:21:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-28T13:21:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-28T13:21:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:00.482 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:00.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-01-28T13:21:01.332 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:34 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:01.492 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:01.493 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-01-28T13:21:01.493 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:21:01.493 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-01-28T13:21:01.493 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:01.493 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 133 k 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-01-28T13:21:01.494 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:01.495 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.8 M 2024-01-28T13:21:01.495 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2024-01-28T13:21:01.495 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2024-01-28T13:21:01.811 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): ceph-mgr-cephadm-19.0.0-933.g00c0d54d.el 573 kB/s | 133 kB 00:00 2024-01-28T13:21:01.894 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-28T13:21:02.353 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 107 kB/s | 58 kB 00:00 2024-01-28T13:21:02.436 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 103 kB/s | 51 kB 00:00 2024-01-28T13:21:02.620 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 626 kB/s | 548 kB 00:00 2024-01-28T13:21:02.624 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:02.624 INFO:teuthology.orchestra.run.smithi019.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-01-28T13:21:02.625 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:02.635 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:02.635 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:02.731 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:02.732 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:03.054 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-28T13:21:03.272 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-28T13:21:03.829 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-28T13:21:03.909 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-28T13:21:03.970 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-28T13:21:04.000 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 6/6 2024-01-28T13:21:04.842 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 6/6 2024-01-28T13:21:04.842 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 1/6 2024-01-28T13:21:04.842 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-28T13:21:04.842 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-28T13:21:04.842 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-28T13:21:04.842 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-28T13:21:05.163 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-28T13:21:05.163 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:05.163 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:05.163 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:21:05.163 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-28T13:21:05.163 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-28T13:21:05.164 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-28T13:21:05.164 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-28T13:21:05.164 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-28T13:21:05.164 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:05.164 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:05.410 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-01-28T13:21:05.998 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:39 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:06.159 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 858 k 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-01-28T13:21:06.160 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:06.161 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 937 k 2024-01-28T13:21:06.161 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 3.1 M 2024-01-28T13:21:06.161 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:06.280 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 661 kB/s | 79 kB 00:00 2024-01-28T13:21:06.605 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-19.0.0-933.g00c0d54d.el9.x86_6 1.9 MB/s | 858 kB 00:00 2024-01-28T13:21:06.606 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:06.607 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.0 MB/s | 937 kB 00:00 2024-01-28T13:21:06.607 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:06.629 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:06.629 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:06.707 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:06.707 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:06.893 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:07.037 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-28T13:21:07.116 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 2/2 2024-01-28T13:21:07.834 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 2/2 2024-01-28T13:21:07.834 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 1/2 2024-01-28T13:21:08.194 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-28T13:21:08.194 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:08.194 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:08.194 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-28T13:21:08.194 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:08.194 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:08.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2024-01-28T13:21:09.043 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:42 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.0.0-933.g00c0d54d.el9 ceph-noarch 263 k 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:21:09.204 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 332 k 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.5 M 2024-01-28T13:21:09.205 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:09.360 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 446 kB/s | 69 kB 00:00 2024-01-28T13:21:09.910 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-volume-19.0.0-933.g00c0d54d.el9.noa 374 kB/s | 263 kB 00:00 2024-01-28T13:21:09.911 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:09.911 INFO:teuthology.orchestra.run.smithi019.stdout:Total 470 kB/s | 332 kB 00:00 2024-01-28T13:21:09.912 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:09.922 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:09.923 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:09.966 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:09.966 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:10.123 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:10.314 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-01-28T13:21:10.356 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.0.0-933.g00c0d54d.el9.noarch 2/2 2024-01-28T13:21:10.372 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.0.0-933.g00c0d54d.el9.noarch 2/2 2024-01-28T13:21:10.372 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:21:10.372 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-28T13:21:10.372 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:11.099 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.0.0-933.g00c0d54d.el9.noarch 1/2 2024-01-28T13:21:11.425 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-01-28T13:21:11.425 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:11.426 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:11.426 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.0.0-933.g00c0d54d.el9.noarch 2024-01-28T13:21:11.426 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-01-28T13:21:11.426 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:11.426 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:11.745 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-01-28T13:21:12.344 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:45 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:12.504 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 127 k 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:21:12.505 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:12.506 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2024-01-28T13:21:12.506 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 457 k 2024-01-28T13:21:12.506 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:12.734 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-19.0.0-933.g00c0d54d.el9.x86_64. 556 kB/s | 127 kB 00:00 2024-01-28T13:21:12.734 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:12.734 INFO:teuthology.orchestra.run.smithi019.stdout:Total 553 kB/s | 127 kB 00:00 2024-01-28T13:21:12.735 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:12.740 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:12.740 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:12.802 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:12.802 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:12.950 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:13.051 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:13.820 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:13.820 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:13.820 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:13.820 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:21:13.820 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:13.820 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:14.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-01-28T13:21:14.658 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:47 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:14.763 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:14.818 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:14.819 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:14.819 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:14.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-01-28T13:21:15.485 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:48 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:15.644 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 31 k 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:15.645 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:15.646 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:15.646 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:21:15.646 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:15.646 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 k 2024-01-28T13:21:15.646 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 140 k 2024-01-28T13:21:15.646 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:15.903 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-19.0.0-933.g00c0d54d.el9.x86_64 121 kB/s | 31 kB 00:00 2024-01-28T13:21:15.903 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:15.904 INFO:teuthology.orchestra.run.smithi019.stdout:Total 121 kB/s | 31 kB 00:00 2024-01-28T13:21:15.904 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:15.907 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:15.907 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:15.939 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:15.940 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:16.037 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:16.153 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:16.767 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:16.767 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:16.767 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:16.767 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:21:16.767 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:16.767 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:16.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-01-28T13:21:17.516 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:50 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:17.622 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:17.676 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:17.677 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:17.677 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:17.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-01-28T13:21:18.337 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:51 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:18.441 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:18.496 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:18.497 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:18.497 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:18.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-01-28T13:21:19.158 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:52 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:19.262 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:19.316 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:19.317 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:19.317 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:19.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-01-28T13:21:19.977 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:52 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:20.084 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:20.139 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:20.140 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:20.140 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:20.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-01-28T13:21:20.808 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:53 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:20.912 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:20.967 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:20.969 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:20.969 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:21.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-01-28T13:21:21.627 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:54 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:21.732 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:19.0.0-933.g00c0d54d.el9.x86_64 is already installed. 2024-01-28T13:21:21.786 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:21.787 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-28T13:21:21.788 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:21.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-01-28T13:21:22.448 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:55 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:22.608 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:22.609 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:22.609 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-01-28T13:21:22.609 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:22.609 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:22.609 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 86 k 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 86 k 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 256 k 2024-01-28T13:21:22.610 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:22.814 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-19.0.0-933.g00c0d54d.el9.x86_64.rpm 420 kB/s | 86 kB 00:00 2024-01-28T13:21:22.815 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:22.815 INFO:teuthology.orchestra.run.smithi019.stdout:Total 418 kB/s | 86 kB 00:00 2024-01-28T13:21:22.815 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:22.822 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:22.822 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:22.879 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:22.879 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:23.016 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:23.115 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:23.524 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:23.829 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:23.829 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:23.830 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:23.830 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:21:23.830 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:23.830 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:24.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-01-28T13:21:24.630 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:57 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:24.790 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 3.0 M 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:24.791 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:21:24.792 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:24.792 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.0 M 2024-01-28T13:21:24.792 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 14 M 2024-01-28T13:21:24.792 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:25.227 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-19.0.0-933.g00c0d54d.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-01-28T13:21:25.227 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:25.227 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.0 MB/s | 3.0 MB 00:00 2024-01-28T13:21:25.227 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:25.236 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:25.237 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:25.312 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:25.312 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:25.764 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:25.835 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:25.858 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:25.859 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-28T13:21:25.859 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-28T13:21:25.859 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-28T13:21:25.859 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-28T13:21:25.859 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:26.990 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:26.991 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:26.991 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:26.991 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:21:26.991 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:26.991 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:27.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-01-28T13:21:27.793 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:00 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:21:27.953 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 169 k 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:21:27.954 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:21:27.955 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-01-28T13:21:27.955 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:27.955 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 169 k 2024-01-28T13:21:27.955 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 540 k 2024-01-28T13:21:27.955 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:21:28.297 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-19.0.0-933.g00c0d54d.el9.x86_64.rpm 495 kB/s | 169 kB 00:00 2024-01-28T13:21:28.297 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:21:28.297 INFO:teuthology.orchestra.run.smithi019.stdout:Total 493 kB/s | 169 kB 00:00 2024-01-28T13:21:28.298 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:21:28.305 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:21:28.306 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:21:28.362 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:21:28.363 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:21:28.521 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:21:28.622 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:29.065 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:29.389 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.0.0-933.g00c0d54d.el9.x86_64 1/1 2024-01-28T13:21:29.389 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:29.389 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:21:29.389 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:21:29.389 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:29.389 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:21:29.629 DEBUG:teuthology.parallel:result is None 2024-01-28T13:21:29.630 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=00c0d54d57237e78360214c1fefe250a2610dcfe 2024-01-28T13:21:29.858 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-28T13:21:29.879 INFO:teuthology.orchestra.run.smithi019.stdout:19.0.0-933.g00c0d54d.el9 2024-01-28T13:21:29.879 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-933.g00c0d54d.el9 2024-01-28T13:21:29.879 INFO:teuthology.task.install:The correct ceph version 19.0.0-933.g00c0d54d is installed. 2024-01-28T13:21:29.881 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-28T13:21:29.881 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:29.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-28T13:21:29.947 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-28T13:21:29.948 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:29.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-01-28T13:21:30.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-28T13:21:30.081 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-28T13:21:30.081 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:30.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-28T13:21:30.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-28T13:21:30.214 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-28T13:21:30.214 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:30.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-01-28T13:21:30.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-28T13:21:30.349 INFO:teuthology.run_tasks:Running task ceph... 2024-01-28T13:21:30.436 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-28T13:21:30.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-01-28T13:21:30.463 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-28T13:21:30.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-28T13:21:30.529 INFO:tasks.ceph:Creating extra log directories... 2024-01-28T13:21:30.529 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-28T13:21:30.603 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-28T13:21:30.603 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-28T13:21:30.603 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830', 'branch': 'wip-matanb-crimson-pg-map-logs', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio}', 'email': None, 'first_in_suite': False, 'job_id': '7535830', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-pg-map-logs'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '7eb9e33f53ab64f23bcf0a2a31509f3442110e4e'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 7326, 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_7eb9e33f53ab64f23bcf0a2a31509f3442110e4e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '7eb9e33f53ab64f23bcf0a2a31509f3442110e4e', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSk5Mv474VqVpbJCvzOlBHvdy1h1UC2aszlUiSuKbYnM4+RNzTOSbkTL3ClL7pfs1+8iov/Lmh5srCg+0HgsbE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '00c0d54d57237e78360214c1fefe250a2610dcfe', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c856e0f9244dd29a50697123b480d63f16a4ed81', 'timestamp': '2024-01-28_12:38:29', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-28T13:21:30.604 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-28T13:21:30.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-01-28T13:21:30.717 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:30.717 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-01-28T13:21:30.772 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-28T13:21:30.772 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-28 13:21:26.118646688 +0000 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-28 13:17:41.893868571 +0000 2024-01-28T13:21:30.826 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-28 13:17:41.893868571 +0000 2024-01-28T13:21:30.827 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-01-28 13:17:41.893868571 +0000 2024-01-28T13:21:30.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-28T13:21:30.891 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-28T13:21:30.891 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-28T13:21:30.892 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000196525 s, 2.6 MB/s 2024-01-28T13:21:30.893 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-28T13:21:30.948 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-28 13:21:26.118646688 +0000 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-28 13:17:42.227865825 +0000 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-28 13:17:42.227865825 +0000 2024-01-28T13:21:31.002 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-01-28 13:17:42.227865825 +0000 2024-01-28T13:21:31.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-28T13:21:31.067 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-28T13:21:31.067 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-28T13:21:31.067 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000188863 s, 2.7 MB/s 2024-01-28T13:21:31.068 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-28T13:21:31.124 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-28 13:21:26.118646688 +0000 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-28 13:17:42.575862964 +0000 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-28 13:17:42.575862964 +0000 2024-01-28T13:21:31.178 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-01-28 13:17:42.575862964 +0000 2024-01-28T13:21:31.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-28T13:21:31.243 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-28T13:21:31.243 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-28T13:21:31.243 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000207733 s, 2.5 MB/s 2024-01-28T13:21:31.244 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-28T13:21:31.299 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-01-28T13:21:31.353 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-28T13:21:31.353 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-28 13:21:26.118646688 +0000 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-28 13:17:42.916860160 +0000 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-28 13:17:42.916860160 +0000 2024-01-28T13:21:31.354 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-01-28 13:17:42.916860160 +0000 2024-01-28T13:21:31.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-28T13:21:31.418 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-28T13:21:31.418 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-28T13:21:31.418 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000171867 s, 3.0 MB/s 2024-01-28T13:21:31.420 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-28T13:21:31.475 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-28T13:21:31.475 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-28T13:21:31.475 INFO:tasks.ceph:Generating config... 2024-01-28T13:21:31.476 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-28T13:21:31.476 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] err to stderr = True 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] log flush on exit = True 2024-01-28T13:21:31.477 INFO:tasks.ceph:[osd] log to file = False 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] log to stderr = True 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-28T13:21:31.478 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] auth client required = none 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] auth cluster required = none 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] auth service required = none 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] auth supported = none 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] debug asok = 0/0 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] debug auth = 0/0 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-01-28T13:21:31.478 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug client = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug context = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug crush = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug filer = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug journal = 0/0 2024-01-28T13:21:31.479 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mds = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug mon = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug monc = 0/0 2024-01-28T13:21:31.480 INFO:tasks.ceph:[global] debug ms = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug osd = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rados = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug refs = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-01-28T13:21:31.481 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] debug striper = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] debug timer = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] debug tp = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] ms client mode = crc 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-01-28T13:21:31.482 INFO:tasks.ceph:[global] ms service mode = crc 2024-01-28T13:21:31.482 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-28T13:21:31.482 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-28T13:21:31.482 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-01-28T13:21:31.482 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-01-28T13:21:31.483 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-01-28T13:21:31.483 INFO:tasks.ceph:Setting up mon.a... 2024-01-28T13:21:31.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-28T13:21:31.571 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2024-01-28T13:21:31.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-28T13:21:31.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-28T13:21:31.648 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.19')] 2024-01-28T13:21:31.649 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-01-28T13:21:31.650 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:31.650 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-28T13:21:31.703 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:31.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:31.802+0000 7f62be1d7e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:31.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:31.803+0000 7f62be1d7e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:31.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:31.803+0000 7f62be1d7e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 391169b2-0530-4262-badc-964afa59a213 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 391169b2-0530-4262-badc-964afa59a213 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-01-28T13:21:31.804434+0000 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-01-28T13:21:31.804434+0000 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 17 (quincy) 2024-01-28T13:21:31.789 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2024-01-28T13:21:31.790 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-01-28T13:21:31.790 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-28T13:21:31.791 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-28T13:21:31.805 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 391169b2-0530-4262-badc-964afa59a213... 2024-01-28T13:21:31.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-28T13:21:31.894 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-01-28T13:21:31.894 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-01-28T13:21:31.894 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-28T13:21:31.894 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = none 2024-01-28T13:21:31.894 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-28T13:21:31.895 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-01-28T13:21:31.896 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.19 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: auth client required = none 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: auth cluster required = none 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: auth service required = none 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug asok = 0/0 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 0/0 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug bdev = 0/0 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 0/0 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 0/0 2024-01-28T13:21:31.897 INFO:teuthology.orchestra.run.smithi019.stdout: debug buffer = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug civetweb = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug client = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug compressor = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug context = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug crush = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug crypto = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug dpdk = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug eventtrace = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug filer = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug filestore = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug finisher = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug fuse = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug heartbeatmap = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug javaclient = 0/0 2024-01-28T13:21:31.898 INFO:teuthology.orchestra.run.smithi019.stdout: debug journal = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug journaler = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug kstore = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug leveldb = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug lockdep = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds balancer = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds locker = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds log = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds log expire = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds migrator = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug memdb = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgrc = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2024-01-28T13:21:31.899 INFO:teuthology.orchestra.run.smithi019.stdout: debug objclass = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug objectcacher = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug objecter = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug optracker = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug perfcounter = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rados = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd mirror = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd replay = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug refs = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw sync = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 0/0 2024-01-28T13:21:31.900 INFO:teuthology.orchestra.run.smithi019.stdout: debug striper = 0/0 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: debug throttle = 0/0 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: debug timer = 0/0 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: debug tp = 0/0 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: ms client mode = crc 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: ms cluster mode = crc 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon client mode = crc 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon cluster mode = crc 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon service mode = crc 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: ms service mode = crc 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 391169b2-0530-4262-badc-964afa59a213 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.901 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-01-28T13:21:31.902 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression algorithm = snappy 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression mode = aggressive 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore rocksdb cf = False 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: crimson osd obc lru size = 10 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: crimson seastar smp = 3 2024-01-28T13:21:31.903 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 20 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 20 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: debug filestore = 0/0 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: debug journal = 0/0 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 0/0 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 10 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: err to stderr = True 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: log flush on exit = True 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: log to file = False 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: log to stderr = True 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: osd shutdown pgref assert = True 2024-01-28T13:21:31.904 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 0/0 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 0/0 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-01-28T13:21:31.905 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-01-28T13:21:31.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default crimson = True 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-28T13:21:31.907 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-01-28T13:21:31.908 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-28T13:21:31.908 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-28T13:21:31.968 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-28T13:21:31.969 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:31.969 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-28T13:21:31.983 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:31.983 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-28T13:21:32.038 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:21:32.038 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:32.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-28T13:21:32.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-28T13:21:32.117 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:32.117 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:32.171 INFO:tasks.ceph:Setting up mon nodes... 2024-01-28T13:21:32.171 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-28T13:21:32.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-28T13:21:32.271 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-28T13:21:32.273 INFO:tasks.ceph:Setting up mds nodes... 2024-01-28T13:21:32.273 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-28T13:21:32.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-28T13:21:32.321 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-28T13:21:32.336 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-28T13:21:32.337 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-28T13:21:32.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-28T13:21:32.363 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-28T13:21:32.363 INFO:tasks.ceph:role: osd.0 2024-01-28T13:21:32.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:21:32.363 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-28T13:21:32.436 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-28T13:21:32.436 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-28T13:21:32.436 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-28T13:21:32.436 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-28T13:21:32.437 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-28T13:21:32.437 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-01-28T13:21:32.437 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-28T13:21:32.437 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-28T13:21:32.437 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-28T13:21:32.437 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-28T13:21:32.952 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-01-28T13:21:32.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-01-28T13:21:32.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-28T13:21:33.094 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-28T13:21:33.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-28T13:21:33.197 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-28T13:21:33.198 INFO:tasks.ceph:role: osd.1 2024-01-28T13:21:33.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:21:33.198 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-28T13:21:33.273 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-28T13:21:33.722 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-01-28T13:21:33.723 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-01-28T13:21:33.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-28T13:21:33.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-28T13:21:33.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-28T13:21:33.920 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-28T13:21:33.920 INFO:tasks.ceph:role: osd.2 2024-01-28T13:21:33.920 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:21:33.920 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-28T13:21:33.995 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-28T13:21:34.505 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-01-28T13:21:34.507 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-01-28T13:21:34.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-28T13:21:34.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-28T13:21:34.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:35.247 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-28T13:21:35.248 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,264 seastar - Reactor backend: linux-aio 2024-01-28T13:21:35.263 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,279 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-28T13:21:35.263 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,279 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-28T13:21:35.264 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,279 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-28T13:21:35.264 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,279 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-28T13:21:35.266 INFO:teuthology.orchestra.run.smithi019.stderr:==28268==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-28T13:21:35.272 INFO:teuthology.orchestra.run.smithi019.stderr:==28268==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffef50a1000; bottom 0x7f4b98243000; size: 0x00b35ce5e000 (770357714944) 2024-01-28T13:21:35.272 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2024-01-28T13:21:35.272 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-28T13:21:35.826 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:35.827 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2024-01-28T13:21:35.827 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2024-01-28T13:21:35.827 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2024-01-28T13:21:35.827 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:35.827 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:21:35.889 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-28T13:21:35.890 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,906 seastar - Reactor backend: linux-aio 2024-01-28T13:21:35.917 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,933 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-28T13:21:35.918 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,933 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-28T13:21:35.918 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,933 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-28T13:21:35.925 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:35,941 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-28T13:21:35.927 INFO:teuthology.orchestra.run.smithi019.stderr:==28266==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-28T13:21:35.933 INFO:teuthology.orchestra.run.smithi019.stderr:==28266==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffef50a1000; bottom 0x7f4b9602c000; size: 0x00b35f075000 (770393460736) 2024-01-28T13:21:35.934 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2024-01-28T13:21:35.934 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-28T13:21:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8cd756 0x2e8ceb04 0x2e8d0036 0x2e8d03ba 0x2e8d0504 0x2e8d095a 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2786f524 0x2787474c 0x27876fb1 0x27693d7f 0x1fa2ba0e 0x1f792c19 0x1f8bdd1d 0x2ec782df 2024-01-28T13:21:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb2dbe985 0xffffffffb3876437 0xffffffffb3a000ea 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:21:39.961 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 391169b2-0530-4262-badc-964afa59a213 2024-01-28T13:21:40.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-28T13:21:40.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:40.555 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-28T13:21:40.555 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:40,571 seastar - Reactor backend: linux-aio 2024-01-28T13:21:40.571 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:40,587 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-28T13:21:40.571 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:40,587 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-28T13:21:40.572 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:40,587 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-28T13:21:40.572 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:40,587 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-28T13:21:40.573 INFO:teuthology.orchestra.run.smithi019.stderr:==28450==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-28T13:21:40.580 INFO:teuthology.orchestra.run.smithi019.stderr:==28450==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff977d4000; bottom 0x7ffb13968000; size: 0x000483e6c000 (19392806912) 2024-01-28T13:21:40.580 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2024-01-28T13:21:40.580 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-28T13:21:40.606 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8cd756 0x2e8ceb04 0x2e8d0036 0x2e8d03ba 0x2e8d0504 0x2e8d095a 0x54daf 0xc6f8f 0x2c74b 0x2857c 0xb6323 0x240c14d3 0x240c2ce5 0x240c734e 0x240c9172 0x2417bf48 0x2417ccb0 0x242aebf2 0x2436d0c4 0x2436f0db 0x241d9de8 0x241dab46 0x2415d401 0x2415ca70 0x23f502b5 0x25f3afa4 0x25f42ad8 0x25f45425 0x25f465ea 0x23a10523 0x23a6c164 0x23a9dc01 0x2e84010f 0x2e8a3340 0x2eb49287 0x2eb4b0d1 0x2e4fc60e 0x2e4fe517 0x1f9b7275 0x1f9bdd1d 0x1f796b05 0x3feaf 0x3ff5f 0x1f357ba4 2024-01-28T13:21:40.606 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2024-01-28T13:21:41.109 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:41.109 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2024-01-28T13:21:41.109 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2024-01-28T13:21:41.109 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2024-01-28T13:21:41.109 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:41.109 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:21:41.165 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-28T13:21:41.166 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:41,182 seastar - Reactor backend: linux-aio 2024-01-28T13:21:41.184 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:41,200 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-28T13:21:41.184 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:41,200 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-28T13:21:41.184 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:41,200 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-28T13:21:41.191 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:41,207 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-28T13:21:41.194 INFO:teuthology.orchestra.run.smithi019.stderr:==28448==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-28T13:21:41.200 INFO:teuthology.orchestra.run.smithi019.stderr:==28448==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff977d4000; bottom 0x7ffb11751000; size: 0x000486083000 (19428552704) 2024-01-28T13:21:41.200 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2024-01-28T13:21:41.200 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-28T13:21:41.226 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8cd756 0x2e8ceb04 0x2e8d0036 0x2e8d03ba 0x2e8d0504 0x2e8d095a 0x54daf 0x943fc 0x94d46 0x23e0821e 0x23ed0c0b 0x23ed3e1b 0x23f0f1a7 0x23f12673 0x23f12cde 0x23f12e28 0x25f3e341 0x25f42ad8 0x25f45425 0x25f465ea 0x23a10523 0x23a6c164 0x23a9dc01 0x2e84010f 0x2e8a3340 0x2eb49287 0x2eb4b0d1 0x2e4fc60e 0x2e4fe517 0x1f7983bb 0x3feaf 0x3ff5f 0x1f357ba4 2024-01-28T13:21:41.226 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2024-01-28T13:21:45.214 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:45.214 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2024-01-28T13:21:45.214 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2024-01-28T13:21:45.214 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2024-01-28T13:21:45.214 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:45.214 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:21:45.215 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 391169b2-0530-4262-badc-964afa59a213 2024-01-28T13:21:45.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-28T13:21:45.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:45.819 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-28T13:21:45.819 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:45,835 seastar - Reactor backend: linux-aio 2024-01-28T13:21:45.840 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:45,855 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-28T13:21:45.840 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:45,855 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-28T13:21:45.840 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:45,855 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-28T13:21:45.840 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:45,855 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-28T13:21:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:==28632==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-28T13:21:45.848 INFO:teuthology.orchestra.run.smithi019.stderr:==28632==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffec5ab7000; bottom 0x7fa68ab88000; size: 0x00583af2f000 (378946121728) 2024-01-28T13:21:45.848 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2024-01-28T13:21:45.848 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-28T13:21:45.874 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8cd756 0x2e8ceb04 0x2e8d0036 0x2e8d03ba 0x2e8d0504 0x2e8d095a 0x54daf 0x24194813 0x24196589 0x24197c72 0x241df53d 0x241dfa1b 0x24170f65 0x24179b36 0x2417b215 0x2417bbea 0x2417ccb0 0x242aebf2 0x2436d0c4 0x2436f0db 0x241d9de8 0x241dab46 0x2415d401 0x2415ca70 0x23f502b5 0x25f3afa4 0x25f42ad8 0x25f45425 0x25f465ea 0x23a10523 0x23a6c164 0x23a9dc01 0x2e84010f 0x2e8a3340 0x2eb49287 0x2eb4b0d1 0x2e4fc60e 0x2e4fe517 0x1f9b7275 0x1f9bdd1d 0x1f796b05 0x3feaf 0x3ff5f 0x1f357ba4 2024-01-28T13:21:45.874 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2024-01-28T13:21:46.373 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:46.373 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2024-01-28T13:21:46.373 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2024-01-28T13:21:46.373 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2024-01-28T13:21:46.373 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:46.373 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:21:46.428 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-28T13:21:46.429 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:46,445 seastar - Reactor backend: linux-aio 2024-01-28T13:21:46.447 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:46,463 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-28T13:21:46.447 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:46,463 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-28T13:21:46.447 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:46,463 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-28T13:21:46.454 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2024-01-28 13:21:46,470 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-28T13:21:46.457 INFO:teuthology.orchestra.run.smithi019.stderr:==28630==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-28T13:21:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:==28630==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffec5ab7000; bottom 0x7fa688971000; size: 0x00583d146000 (378981867520) 2024-01-28T13:21:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2024-01-28T13:21:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-28T13:21:46.489 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8cd756 0x2e8ceb04 0x2e8d0036 0x2e8d03ba 0x2e8d0504 0x2e8d095a 0x54daf 0x39769 0x94456 0x94d46 0x23e07ba7 0x23e07ecf 0x23ed0c0b 0x23ed3e1b 0x23f0f1a7 0x23f14399 0x23f226ee 0x25f3ba5a 0x25f42ad8 0x25f45425 0x25f465ea 0x23a10523 0x23a6c164 0x23a9dc01 0x2e84010f 0x2e8a3340 0x2eb49287 0x2eb4b0d1 0x2e4fc60e 0x2e4fe517 0x1f7983bb 0x3feaf 0x3ff5f 0x1f357ba4 2024-01-28T13:21:46.489 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2024-01-28T13:21:46.548 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8cd756 0x2e8ceb04 0x2e8d0036 0x2e8d03ba 0x2e8d0504 0x2e8d095a 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2786f524 0x2787474c 0x27876fb1 0x27693d7f 0x1fa2ba0e 0x1f792c19 0x1f8bdd1d 0x2ec782df 2024-01-28T13:21:46.548 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb3885943 0xffffffffb3885dcd 0xffffffffb2d522ba 0xffffffffb387645c 0xffffffffb3a000ea 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:21:50.488 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 391169b2-0530-4262-badc-964afa59a213 2024-01-28T13:21:50.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-28T13:21:50.554 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-28T13:21:50.554 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:50.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-28T13:21:50.621 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:50.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-28T13:21:50.688 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:50.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-28T13:21:50.753 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:50.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-28T13:21:50.820 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:50.820 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-28T13:21:50.874 INFO:tasks.ceph:Adding keys to all mons... 2024-01-28T13:21:50.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-28T13:21:50.939 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-01-28T13:21:50.939 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDcVLZlpPsaERAAI9JLfKxJTWSVFgYuGvknPA== 2024-01-28T13:21:50.939 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-01-28T13:21:50.939 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDfVLZl0S+EOxAAnVzGNUYUk7lsChAozdeMZQ== 2024-01-28T13:21:50.939 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-01-28T13:21:50.940 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDlVLZlnDbEDxAAyPEzPStIB2EDjxslxbnfGg== 2024-01-28T13:21:50.940 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-01-28T13:21:50.940 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDqVLZlh3+KHxAATVOm03G1wDd5Lgfzp7z6WA== 2024-01-28T13:21:50.940 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-01-28T13:21:50.940 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDcVLZlfzcbFBAAuljFZPKXD3HrZD7k0OKGEQ== 2024-01-28T13:21:50.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-28T13:21:51.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-28T13:21:51.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-28T13:21:51.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-28T13:21:51.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-28T13:21:51.269 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-28T13:21:51.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-28T13:21:51.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-28T13:21:51.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:51.406+0000 7f263445b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:51.406+0000 7f263445b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-28T13:21:51.429 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-28T13:21:51.483 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-28T13:21:51.484 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-28T13:21:51.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-28T13:21:51.526 INFO:tasks.ceph.mon.a:Started 2024-01-28T13:21:51.526 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-28T13:21:51.526 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-28T13:21:51.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-28T13:21:51.528 INFO:tasks.ceph.mgr.x:Started 2024-01-28T13:21:51.528 INFO:tasks.ceph:Setting crush tunables to default 2024-01-28T13:21:51.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2024-01-28T13:21:51.607 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-01-28T13:21:51.622+0000 7f7106a6b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.607 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-01-28T13:21:51.622+0000 7f7106a6b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.607 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-01-28T13:21:51.622+0000 7f7106a6b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.607 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:51.622+0000 7f2582409f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:51.649+0000 7f646e75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:51.678+0000 7f646e75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.734 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-01-28T13:21:51.745 INFO:tasks.ceph:check_enable_crimson: True 2024-01-28T13:21:51.746 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-01-28T13:21:51.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-01-28T13:21:51.810 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:51.825+0000 7f2582409f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.810 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:51.826+0000 7f2582409f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:51.862+0000 7ff4ef39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:51.868+0000 7ff4ef39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:51.936 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-01-28T13:21:51.951+0000 7f70fe9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-01-28T13:21:51.936 INFO:tasks.ceph.mon.a.smithi019.stderr:Please be aware that this feature is experimental, untested, 2024-01-28T13:21:51.937 INFO:tasks.ceph.mon.a.smithi019.stderr:unsupported, and may result in data corruption, data loss, 2024-01-28T13:21:51.937 INFO:tasks.ceph.mon.a.smithi019.stderr:and/or irreparable damage to your cluster. Do not use 2024-01-28T13:21:51.937 INFO:tasks.ceph.mon.a.smithi019.stderr:feature with important data. 2024-01-28T13:21:51.937 INFO:tasks.ceph.mon.a.smithi019.stderr: 2024-01-28T13:21:52.274 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:52.289+0000 7f2582409f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-28T13:21:52.329 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:52.344+0000 7f2582409f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-28T13:21:52.747 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-28T13:21:52.747 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:52.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-28T13:21:52.774 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:52.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-28T13:21:52.840 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:21:52.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-28T13:21:52.877 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:52.892+0000 7f2582409f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-28T13:21:52.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new ea621e70-6faf-4fc8-b0e3-0e2bb5d2ed92 0 2024-01-28T13:21:52.932 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:52.947+0000 7f2582409f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-28T13:21:52.979 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:52.994+0000 7f2582409f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-28T13:21:53.037 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.052+0000 7f2582409f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-28T13:21:53.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.047+0000 7fe266e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.054+0000 7fe266e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.082 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.097+0000 7f2582409f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-28T13:21:53.117 INFO:teuthology.orchestra.run.smithi019.stdout:0 2024-01-28T13:21:53.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new 30188d5b-3d6f-4b6c-b233-6399b1cb0f5b 1 2024-01-28T13:21:53.189 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.204+0000 7f2582409f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-28T13:21:53.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.243+0000 7f8a02d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.250+0000 7f8a02d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.281 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.296+0000 7f2582409f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-28T13:21:53.329 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.344+0000 7f2582409f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-28T13:21:53.333 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-01-28T13:21:53.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new e66f63ab-5d85-4fcd-9b58-013ee877ac29 2 2024-01-28T13:21:53.384 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.399+0000 7f2582409f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-28T13:21:53.435 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.450+0000 7f2582409f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-28T13:21:53.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.460+0000 7f8c8bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.467+0000 7f8c8bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.529 INFO:teuthology.orchestra.run.smithi019.stdout:2 2024-01-28T13:21:53.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-01-28T13:21:53.611 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-28T13:21:53.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-01-28T13:21:53.654 INFO:tasks.ceph.osd.0:Started 2024-01-28T13:21:53.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-01-28T13:21:53.684 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-28T13:21:53.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-01-28T13:21:53.727 INFO:tasks.ceph.osd.1:Started 2024-01-28T13:21:53.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-01-28T13:21:53.764 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-28T13:21:53.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-01-28T13:21:53.806 INFO:tasks.ceph.osd.2:Started 2024-01-28T13:21:53.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-28T13:21:53.859 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:53.874+0000 7f2582409f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-28T13:21:53.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.972+0000 7ff31af9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:53.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:21:53.980+0000 7ff31af9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:54.037 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:21:54.037 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":6,"fsid":"391169b2-0530-4262-badc-964afa59a213","created":"2024-01-28T13:21:51.673074+0000","modified":"2024-01-28T13:21:53.545185+0000","last_up_change":"0.000000","last_in_change":"2024-01-28T13:21:53.545185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"ea621e70-6faf-4fc8-b0e3-0e2bb5d2ed92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30188d5b-3d6f-4b6c-b233-6399b1cb0f5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e66f63ab-5d85-4fcd-9b58-013ee877ac29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-28T13:21:54.055 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-28T13:21:54.055 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-28T13:21:54.115 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.130+0000 7f2582409f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-28T13:21:54.261 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.276+0000 7f2582409f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-28T13:21:54.310 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.325+0000 7f2582409f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-28T13:21:54.358 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-28T13:21:54.418 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.433+0000 7f2582409f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-28T13:21:54.456 INFO:teuthology.misc.health.smithi019.stderr:2024-01-28T13:21:54.468+0000 7ff8e9f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:54.457 INFO:teuthology.misc.health.smithi019.stderr:2024-01-28T13:21:54.472+0000 7ff8e9f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:21:54.523 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.538+0000 7f2582409f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-28T13:21:54.566 INFO:teuthology.misc.health.smithi019.stdout: 2024-01-28T13:21:54.566 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":6,"fsid":"391169b2-0530-4262-badc-964afa59a213","created":"2024-01-28T13:21:51.673074+0000","modified":"2024-01-28T13:21:53.545185+0000","last_up_change":"0.000000","last_in_change":"2024-01-28T13:21:53.545185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"ea621e70-6faf-4fc8-b0e3-0e2bb5d2ed92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30188d5b-3d6f-4b6c-b233-6399b1cb0f5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e66f63ab-5d85-4fcd-9b58-013ee877ac29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-28T13:21:54.581 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-28T13:21:54.675 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.690+0000 7f2582409f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-28T13:21:54.721 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.736+0000 7f2582409f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-28T13:21:54.875 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:54.890+0000 7f2582409f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-28T13:21:55.728 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:55.743+0000 7f2582409f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-28T13:21:56.016 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:56.031+0000 7f2582409f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-28T13:21:56.071 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:56.085+0000 7f2582409f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-28T13:21:56.176 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:56.191+0000 7f2582409f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-28T13:21:57.196 INFO:tasks.ceph.mgr.x.smithi019.stderr:2024-01-28T13:21:57.211+0000 7f2522655640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-28T13:22:00.884 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-28T13:22:00.993 INFO:teuthology.misc.health.smithi019.stderr:2024-01-28T13:22:01.005+0000 7f3992ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-01-28T13:22:01.010+0000 7f3992ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:01.182 INFO:teuthology.misc.health.smithi019.stdout: 2024-01-28T13:22:01.182 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":8,"fsid":"391169b2-0530-4262-badc-964afa59a213","created":"2024-01-28T13:21:51.673074+0000","modified":"2024-01-28T13:21:59.211694+0000","last_up_change":"2024-01-28T13:21:59.211694+0000","last_in_change":"2024-01-28T13:21:53.545185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"ea621e70-6faf-4fc8-b0e3-0e2bb5d2ed92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":1083597374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":1083597374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":1083597374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":1083597374}]},"public_addr":"172.21.15.19:6802/1083597374","cluster_addr":"172.21.15.19:6803/1083597374","heartbeat_back_addr":"172.21.15.19:6808/1083597374","heartbeat_front_addr":"172.21.15.19:6809/1083597374","state":["exists","up"]},{"osd":1,"uuid":"30188d5b-3d6f-4b6c-b233-6399b1cb0f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":342911703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":342911703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":342911703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":342911703}]},"public_addr":"172.21.15.19:6804/342911703","cluster_addr":"172.21.15.19:6805/342911703","heartbeat_back_addr":"172.21.15.19:6811/342911703","heartbeat_front_addr":"172.21.15.19:6810/342911703","state":["exists","up"]},{"osd":2,"uuid":"e66f63ab-5d85-4fcd-9b58-013ee877ac29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":1899076272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":1899076272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":1899076272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":1899076272}]},"public_addr":"172.21.15.19:6806/1899076272","cluster_addr":"172.21.15.19:6807/1899076272","heartbeat_back_addr":"172.21.15.19:6812/1899076272","heartbeat_front_addr":"172.21.15.19:6813/1899076272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-28T13:22:01.193 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-28T13:22:01.193 INFO:tasks.ceph:Creating RBD pool 2024-01-28T13:22:01.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-28T13:22:01.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:01.311+0000 7f12d3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:01.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:01.319+0000 7f12d3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'rbd' created 2024-01-28T13:22:02.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-28T13:22:02.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:02.356+0000 7f959d55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:02.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:02.368+0000 7f959d55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:03.215 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-28T13:22:03.229 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-28T13:22:03.229 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-28T13:22:03.230 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-28T13:22:03.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-28T13:22:03.230 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-28T13:22:03.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:03.361+0000 7f3f5575e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:03.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:03.367+0000 7f3f5575e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:03.588 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:03.599 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":3474624489},{"type":"v1","addr":"172.21.15.19:6801","nonce":3474624489}]},"active_addr":"172.21.15.19:6801/3474624489","active_change":"2024-01-28T13:21:56.205748+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":1533332541}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2658062413}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2655567378}]}]} 2024-01-28T13:22:03.603 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-28T13:22:03.603 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-28T13:22:03.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-28T13:22:03.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:03.721+0000 7f970d35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:03.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:03.726+0000 7f970d35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:03.900 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"fsid":"391169b2-0530-4262-badc-964afa59a213","created":"2024-01-28T13:21:51.673074+0000","modified":"2024-01-28T13:22:03.229111+0000","last_up_change":"2024-01-28T13:21:59.211694+0000","last_in_change":"2024-01-28T13:21:53.545185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-28T13:22:00.250049+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-28T13:22:01.523281+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea621e70-6faf-4fc8-b0e3-0e2bb5d2ed92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":1083597374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":1083597374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":1083597374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":1083597374}]},"public_addr":"172.21.15.19:6802/1083597374","cluster_addr":"172.21.15.19:6803/1083597374","heartbeat_back_addr":"172.21.15.19:6808/1083597374","heartbeat_front_addr":"172.21.15.19:6809/1083597374","state":["exists","up"]},{"osd":1,"uuid":"30188d5b-3d6f-4b6c-b233-6399b1cb0f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":342911703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":342911703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":342911703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":342911703}]},"public_addr":"172.21.15.19:6804/342911703","cluster_addr":"172.21.15.19:6805/342911703","heartbeat_back_addr":"172.21.15.19:6811/342911703","heartbeat_front_addr":"172.21.15.19:6810/342911703","state":["exists","up"]},{"osd":2,"uuid":"e66f63ab-5d85-4fcd-9b58-013ee877ac29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":1899076272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":1899076272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":1899076272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":1899076272}]},"public_addr":"172.21.15.19:6806/1899076272","cluster_addr":"172.21.15.19:6807/1899076272","heartbeat_back_addr":"172.21.15.19:6812/1899076272","heartbeat_front_addr":"172.21.15.19:6813/1899076272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-28T13:22:03.913 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-28T13:22:03.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-28T13:22:04.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.032+0000 7f954c31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.037+0000 7f954c31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.207 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:04.208 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"fsid":"391169b2-0530-4262-badc-964afa59a213","created":"2024-01-28T13:21:51.673074+0000","modified":"2024-01-28T13:22:03.229111+0000","last_up_change":"2024-01-28T13:21:59.211694+0000","last_in_change":"2024-01-28T13:21:53.545185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-28T13:22:00.250049+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-28T13:22:01.523281+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea621e70-6faf-4fc8-b0e3-0e2bb5d2ed92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":1083597374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":1083597374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":1083597374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":1083597374}]},"public_addr":"172.21.15.19:6802/1083597374","cluster_addr":"172.21.15.19:6803/1083597374","heartbeat_back_addr":"172.21.15.19:6808/1083597374","heartbeat_front_addr":"172.21.15.19:6809/1083597374","state":["exists","up"]},{"osd":1,"uuid":"30188d5b-3d6f-4b6c-b233-6399b1cb0f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":342911703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":342911703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":342911703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":342911703}]},"public_addr":"172.21.15.19:6804/342911703","cluster_addr":"172.21.15.19:6805/342911703","heartbeat_back_addr":"172.21.15.19:6811/342911703","heartbeat_front_addr":"172.21.15.19:6810/342911703","state":["exists","up"]},{"osd":2,"uuid":"e66f63ab-5d85-4fcd-9b58-013ee877ac29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":1899076272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":1899076272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":1899076272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":1899076272}]},"public_addr":"172.21.15.19:6806/1899076272","cluster_addr":"172.21.15.19:6807/1899076272","heartbeat_back_addr":"172.21.15.19:6812/1899076272","heartbeat_front_addr":"172.21.15.19:6813/1899076272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-28T13:22:04.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-28T13:22:04.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-28T13:22:04.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-28T13:22:04.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.349+0000 7f0d7d59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.350+0000 7f327ab15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.350+0000 7f0c3b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.355+0000 7f0d7d59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.357+0000 7f327ab15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.357+0000 7f0c3b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.366 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-01-28T13:22:04.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-28T13:22:04.367 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-01-28T13:22:04.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-28T13:22:04.369 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-01-28T13:22:04.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-28T13:22:04.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.486+0000 7eff7d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.493+0000 7eff7d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.530+0000 7f8c4235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.532+0000 7f8fda013640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.536+0000 7f8c4235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.538+0000 7f8fda013640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.674 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-01-28T13:22:04.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2024-01-28T13:22:04.687 DEBUG:teuthology.parallel:result is None 2024-01-28T13:22:04.709 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-01-28T13:22:04.717 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-01-28T13:22:04.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2024-01-28T13:22:04.722 DEBUG:teuthology.parallel:result is None 2024-01-28T13:22:04.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2024-01-28T13:22:04.730 DEBUG:teuthology.parallel:result is None 2024-01-28T13:22:04.730 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-28T13:22:04.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-28T13:22:04.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.888+0000 7f741302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:04.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:04.893+0000 7f741302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:05.065 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-01-28T13:22:05.077 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-01-28T13:22:04.371672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":413696,"data_stored":1248195,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.342847+0000","last_change":"2024-01-28T13:22:03.342904+0000","last_active":"2024-01-28T13:22:03.342847+0000","last_peered":"2024-01-28T13:22:03.342847+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.341209+0000","last_became_peered":"2024-01-28T13:22:03.341209+0000","last_unstale":"2024-01-28T13:22:03.342847+0000","last_undegraded":"2024-01-28T13:22:03.342847+0000","last_fullsized":"2024-01-28T13:22:03.342847+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.323227+0000","last_change":"2024-01-28T13:22:03.323381+0000","last_active":"2024-01-28T13:22:03.323227+0000","last_peered":"2024-01-28T13:22:03.323227+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.320630+0000","last_became_peered":"2024-01-28T13:22:03.320630+0000","last_unstale":"2024-01-28T13:22:03.323227+0000","last_undegraded":"2024-01-28T13:22:03.323227+0000","last_fullsized":"2024-01-28T13:22:03.323227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.339708+0000","last_change":"2024-01-28T13:22:03.339765+0000","last_active":"2024-01-28T13:22:03.339708+0000","last_peered":"2024-01-28T13:22:03.339708+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.337931+0000","last_became_peered":"2024-01-28T13:22:03.337931+0000","last_unstale":"2024-01-28T13:22:03.339708+0000","last_undegraded":"2024-01-28T13:22:03.339708+0000","last_fullsized":"2024-01-28T13:22:03.339708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.332903+0000","last_change":"2024-01-28T13:22:03.332998+0000","last_active":"2024-01-28T13:22:03.332903+0000","last_peered":"2024-01-28T13:22:03.332903+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.321612+0000","last_became_peered":"2024-01-28T13:22:03.321612+0000","last_unstale":"2024-01-28T13:22:03.332903+0000","last_undegraded":"2024-01-28T13:22:03.332903+0000","last_fullsized":"2024-01-28T13:22:03.332903+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-28T13:22:04.271629+0000","last_change":"2024-01-28T13:22:03.313613+0000","last_active":"2024-01-28T13:22:04.271629+0000","last_peered":"2024-01-28T13:22:04.271629+0000","last_clean":"2024-01-28T13:22:04.271629+0000","last_became_active":"2024-01-28T13:22:03.311800+0000","last_became_peered":"2024-01-28T13:22:03.311800+0000","last_unstale":"2024-01-28T13:22:04.271629+0000","last_undegraded":"2024-01-28T13:22:04.271629+0000","last_fullsized":"2024-01-28T13:22:04.271629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-28T13:22:04.240802+0000","last_change":"2024-01-28T13:22:03.319261+0000","last_active":"2024-01-28T13:22:04.240802+0000","last_peered":"2024-01-28T13:22:04.240802+0000","last_clean":"2024-01-28T13:22:04.240802+0000","last_became_active":"2024-01-28T13:22:03.317395+0000","last_became_peered":"2024-01-28T13:22:03.317395+0000","last_unstale":"2024-01-28T13:22:04.240802+0000","last_undegraded":"2024-01-28T13:22:04.240802+0000","last_fullsized":"2024-01-28T13:22:04.240802+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-28T13:22:04.240274+0000","last_change":"2024-01-28T13:22:03.308545+0000","last_active":"2024-01-28T13:22:04.240274+0000","last_peered":"2024-01-28T13:22:04.240274+0000","last_clean":"2024-01-28T13:22:04.240274+0000","last_became_active":"2024-01-28T13:22:03.306391+0000","last_became_peered":"2024-01-28T13:22:03.306391+0000","last_unstale":"2024-01-28T13:22:04.240274+0000","last_undegraded":"2024-01-28T13:22:04.240274+0000","last_fullsized":"2024-01-28T13:22:04.240274+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.297942+0000","last_change":"2024-01-28T13:22:03.298013+0000","last_active":"2024-01-28T13:22:03.297942+0000","last_peered":"2024-01-28T13:22:03.297942+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.296270+0000","last_became_peered":"2024-01-28T13:22:03.296270+0000","last_unstale":"2024-01-28T13:22:03.297942+0000","last_undegraded":"2024-01-28T13:22:03.297942+0000","last_fullsized":"2024-01-28T13:22:03.297942+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.257961+0000","last_change":"2024-01-28T13:22:02.301557+0000","last_active":"2024-01-28T13:22:03.257961+0000","last_peered":"2024-01-28T13:22:03.257961+0000","last_clean":"2024-01-28T13:22:03.257961+0000","last_became_active":"2024-01-28T13:22:02.281448+0000","last_became_peered":"2024-01-28T13:22:02.281448+0000","last_unstale":"2024-01-28T13:22:03.257961+0000","last_undegraded":"2024-01-28T13:22:03.257961+0000","last_fullsized":"2024-01-28T13:22:03.257961+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:01.216626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:01.216626+0000","last_clean_scrub_stamp":"2024-01-28T13:22:01.216626+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-28T13:22:05.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-28T13:22:05.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:05.197+0000 7f3b27217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:05.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:05.201+0000 7f3b27217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:05.382 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-01-28T13:22:05.393 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-01-28T13:22:04.371672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":413696,"data_stored":1248195,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.342847+0000","last_change":"2024-01-28T13:22:03.342904+0000","last_active":"2024-01-28T13:22:03.342847+0000","last_peered":"2024-01-28T13:22:03.342847+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.341209+0000","last_became_peered":"2024-01-28T13:22:03.341209+0000","last_unstale":"2024-01-28T13:22:03.342847+0000","last_undegraded":"2024-01-28T13:22:03.342847+0000","last_fullsized":"2024-01-28T13:22:03.342847+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.323227+0000","last_change":"2024-01-28T13:22:03.323381+0000","last_active":"2024-01-28T13:22:03.323227+0000","last_peered":"2024-01-28T13:22:03.323227+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.320630+0000","last_became_peered":"2024-01-28T13:22:03.320630+0000","last_unstale":"2024-01-28T13:22:03.323227+0000","last_undegraded":"2024-01-28T13:22:03.323227+0000","last_fullsized":"2024-01-28T13:22:03.323227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.339708+0000","last_change":"2024-01-28T13:22:03.339765+0000","last_active":"2024-01-28T13:22:03.339708+0000","last_peered":"2024-01-28T13:22:03.339708+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.337931+0000","last_became_peered":"2024-01-28T13:22:03.337931+0000","last_unstale":"2024-01-28T13:22:03.339708+0000","last_undegraded":"2024-01-28T13:22:03.339708+0000","last_fullsized":"2024-01-28T13:22:03.339708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.332903+0000","last_change":"2024-01-28T13:22:03.332998+0000","last_active":"2024-01-28T13:22:03.332903+0000","last_peered":"2024-01-28T13:22:03.332903+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.321612+0000","last_became_peered":"2024-01-28T13:22:03.321612+0000","last_unstale":"2024-01-28T13:22:03.332903+0000","last_undegraded":"2024-01-28T13:22:03.332903+0000","last_fullsized":"2024-01-28T13:22:03.332903+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-28T13:22:04.271629+0000","last_change":"2024-01-28T13:22:03.313613+0000","last_active":"2024-01-28T13:22:04.271629+0000","last_peered":"2024-01-28T13:22:04.271629+0000","last_clean":"2024-01-28T13:22:04.271629+0000","last_became_active":"2024-01-28T13:22:03.311800+0000","last_became_peered":"2024-01-28T13:22:03.311800+0000","last_unstale":"2024-01-28T13:22:04.271629+0000","last_undegraded":"2024-01-28T13:22:04.271629+0000","last_fullsized":"2024-01-28T13:22:04.271629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-28T13:22:04.240802+0000","last_change":"2024-01-28T13:22:03.319261+0000","last_active":"2024-01-28T13:22:04.240802+0000","last_peered":"2024-01-28T13:22:04.240802+0000","last_clean":"2024-01-28T13:22:04.240802+0000","last_became_active":"2024-01-28T13:22:03.317395+0000","last_became_peered":"2024-01-28T13:22:03.317395+0000","last_unstale":"2024-01-28T13:22:04.240802+0000","last_undegraded":"2024-01-28T13:22:04.240802+0000","last_fullsized":"2024-01-28T13:22:04.240802+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-28T13:22:04.240274+0000","last_change":"2024-01-28T13:22:03.308545+0000","last_active":"2024-01-28T13:22:04.240274+0000","last_peered":"2024-01-28T13:22:04.240274+0000","last_clean":"2024-01-28T13:22:04.240274+0000","last_became_active":"2024-01-28T13:22:03.306391+0000","last_became_peered":"2024-01-28T13:22:03.306391+0000","last_unstale":"2024-01-28T13:22:04.240274+0000","last_undegraded":"2024-01-28T13:22:04.240274+0000","last_fullsized":"2024-01-28T13:22:04.240274+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.297942+0000","last_change":"2024-01-28T13:22:03.298013+0000","last_active":"2024-01-28T13:22:03.297942+0000","last_peered":"2024-01-28T13:22:03.297942+0000","last_clean":"2024-01-28T13:22:02.224697+0000","last_became_active":"2024-01-28T13:22:03.296270+0000","last_became_peered":"2024-01-28T13:22:03.296270+0000","last_unstale":"2024-01-28T13:22:03.297942+0000","last_undegraded":"2024-01-28T13:22:03.297942+0000","last_fullsized":"2024-01-28T13:22:03.297942+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:02.224697+0000","last_clean_scrub_stamp":"2024-01-28T13:22:02.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-28T13:22:03.257961+0000","last_change":"2024-01-28T13:22:02.301557+0000","last_active":"2024-01-28T13:22:03.257961+0000","last_peered":"2024-01-28T13:22:03.257961+0000","last_clean":"2024-01-28T13:22:03.257961+0000","last_became_active":"2024-01-28T13:22:02.281448+0000","last_became_peered":"2024-01-28T13:22:02.281448+0000","last_unstale":"2024-01-28T13:22:03.257961+0000","last_undegraded":"2024-01-28T13:22:03.257961+0000","last_fullsized":"2024-01-28T13:22:03.257961+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-28T13:22:01.216626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-28T13:22:01.216626+0000","last_clean_scrub_stamp":"2024-01-28T13:22:01.216626+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-28T13:22:05.394 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-28T13:22:05.394 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-28T13:22:05.394 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-28T13:22:05.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-28T13:22:05.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:05.513+0000 7f2c6bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:05.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:22:05.517+0000 7f2c6bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:22:05.726 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:05.726 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-28T13:22:05.738 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-28T13:22:05.738 INFO:teuthology.run_tasks:Running task cbt... 2024-01-28T13:22:05.751 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi019.front.sepia.ceph.com', 'osds': ['ubuntu@smithi019.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi019.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi019.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi019.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-01-28T13:22:05.751 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-01-28T13:22:05.767 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:22:05.767 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-28T13:22:05.823 DEBUG:teuthology.orchestra.run.smithi019:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-01-28T13:22:05.875 INFO:teuthology.orchestra.run.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-01-28T13:22:06.247 INFO:tasks.cbt:Installing collectl 2024-01-28T13:22:06.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-01-28T13:22:06.351 INFO:teuthology.orchestra.run.smithi019.stderr:--2024-01-28 13:22:06-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-01-28T13:22:06.491 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-01-28T13:22:06.501 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-01-28T13:22:06.725 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-28T13:22:06.725 INFO:teuthology.orchestra.run.smithi019.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABltlT-MdCLk6SJ-yvSuyJq3FjXexhOYrmZ8A48bOKMod7eN0mzkyHchVFkX5LTZOg4n1_IlPTBeA3KzTjLgS9XpATvVw%3D%3D&use_mirror=cytranet&r= [following] 2024-01-28T13:22:06.725 INFO:teuthology.orchestra.run.smithi019.stderr:--2024-01-28 13:22:06-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABltlT-MdCLk6SJ-yvSuyJq3FjXexhOYrmZ8A48bOKMod7eN0mzkyHchVFkX5LTZOg4n1_IlPTBeA3KzTjLgS9XpATvVw%3D%3D&use_mirror=cytranet&r= 2024-01-28T13:22:06.725 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-01-28T13:22:06.796 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-01-28T13:22:07.033 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-28T13:22:07.034 INFO:teuthology.orchestra.run.smithi019.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-01-28T13:22:07.034 INFO:teuthology.orchestra.run.smithi019.stderr:--2024-01-28 13:22:07-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-01-28T13:22:07.044 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2024-01-28T13:22:07.114 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2024-01-28T13:22:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 200 OK 2024-01-28T13:22:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:Length: 620693 (606K) [application/x-gzip] 2024-01-28T13:22:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:Saving to: ‘collectl.tar.gz’ 2024-01-28T13:22:07.439 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:22:07.509 INFO:teuthology.orchestra.run.smithi019.stderr: 0K .......... .......... .......... .......... .......... 8% 710K 1s 2024-01-28T13:22:07.578 INFO:teuthology.orchestra.run.smithi019.stderr: 50K .......... .......... .......... .......... .......... 16% 717K 1s 2024-01-28T13:22:07.580 INFO:teuthology.orchestra.run.smithi019.stderr: 100K .......... .......... .......... .......... .......... 24% 41.3M 0s 2024-01-28T13:22:07.648 INFO:teuthology.orchestra.run.smithi019.stderr: 150K .......... .......... .......... .......... .......... 32% 731K 0s 2024-01-28T13:22:07.651 INFO:teuthology.orchestra.run.smithi019.stderr: 200K .......... .......... .......... .......... .......... 41% 46.2M 0s 2024-01-28T13:22:07.651 INFO:teuthology.orchestra.run.smithi019.stderr: 250K .......... .......... .......... .......... .......... 49% 56.1M 0s 2024-01-28T13:22:07.651 INFO:teuthology.orchestra.run.smithi019.stderr: 300K .......... .......... .......... .......... .......... 57% 44.6M 0s 2024-01-28T13:22:07.722 INFO:teuthology.orchestra.run.smithi019.stderr: 350K .......... .......... .......... .......... .......... 65% 748K 0s 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 400K .......... .......... .......... .......... .......... 74% 75.5M 0s 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 450K .......... .......... .......... .......... .......... 82% 72.5M 0s 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 500K .......... .......... .......... .......... .......... 90% 79.1M 0s 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 550K .......... .......... .......... .......... .......... 98% 82.0M 0s 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 600K ...... 100% 11.4T=0.3s 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28 13:22:07 (2.10 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-01-28T13:22:07.723 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/ 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/collectl.conf 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/SlowProc.html 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/WhySummary.html 2024-01-28T13:22:07.765 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Socket.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/colmux.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Tutorial.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Gexpr.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Import.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/style.css 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/FileNaming.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/TheMath.html 2024-01-28T13:22:07.766 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Environmental.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Process.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/HiResTime.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/slub.jpg 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Lustre.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Documentation.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Inodes.html 2024-01-28T13:22:07.767 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Graphite.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/InputFiles.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Export.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Interrupts.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Startup.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Performance.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Messages.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Logging.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-detail.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/CPUs.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Matrix.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-brief.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Misc.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Examples.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OpenStack.html 2024-01-28T13:22:07.768 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Exceptions.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Playback.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Hello.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Sockets.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Network.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Infiniband.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Memory.html 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-01-28T13:22:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Disks.html 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/service/ 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/service/collectl.service 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/ 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-suse 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-debian 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-generic 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/misc.ph 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/gexpr.ph 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/statsd.ph 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/envrules.std 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/hello.ph 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vmsum.ph 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/proctree.ph 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/README-WINDOWS 2024-01-28T13:22:07.770 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/formatit.ph 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/colmux 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/INSTALL 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/graphite.ph 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/RELEASE-collectl 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vmstat.ph 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/README 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/ 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/collectl.1 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/colmux.1 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/lexpr.ph 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/client.pl 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/collectl 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/COPYING 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vnet.ph 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/GPL 2024-01-28T13:22:07.771 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/UNINSTALL 2024-01-28T13:22:07.772 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/ARTISTIC 2024-01-28T13:22:07.818 INFO:teuthology.orchestra.run.smithi019.stderr:./INSTALL: line 90: chkconfig: command not found 2024-01-28T13:22:07.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-01-28T13:22:08.411 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:41 ago on Sun 28 Jan 2024 01:18:27 PM UTC. 2024-01-28T13:22:08.518 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-01-28T13:22:08.521 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-01-28T13:22:08.579 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-28T13:22:08.580 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:22:08.580 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-28T13:22:08.580 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: librbd-devel x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 23 k 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: perf x86_64 5.14.0-411.el9 CentOS-AppStream 8.0 M 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: libradospp-devel x86_64 2:19.0.0-933.g00c0d54d.el9 ceph 30 k 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout:Install 5 Packages 2024-01-28T13:22:08.581 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:08.582 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 8.2 M 2024-01-28T13:22:08.582 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 11 M 2024-01-28T13:22:08.582 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-28T13:22:09.148 INFO:teuthology.orchestra.run.smithi019.stdout:(1/5): libradospp-devel-19.0.0-933.g00c0d54d.el 154 kB/s | 30 kB 00:00 2024-01-28T13:22:09.290 INFO:teuthology.orchestra.run.smithi019.stdout:(2/5): perf-5.14.0-411.el9.x86_64.rpm 24 MB/s | 8.0 MB 00:00 2024-01-28T13:22:09.315 INFO:teuthology.orchestra.run.smithi019.stdout:(3/5): librbd-devel-19.0.0-933.g00c0d54d.el9.x8 64 kB/s | 23 kB 00:00 2024-01-28T13:22:09.507 INFO:teuthology.orchestra.run.smithi019.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 60 kB/s | 13 kB 00:00 2024-01-28T13:22:09.657 INFO:teuthology.orchestra.run.smithi019.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 234 kB/s | 118 kB 00:00 2024-01-28T13:22:09.659 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-28T13:22:09.659 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.6 MB/s | 8.2 MB 00:01 2024-01-28T13:22:09.659 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-28T13:22:09.682 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-28T13:22:09.683 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-28T13:22:09.853 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-28T13:22:09.857 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-28T13:22:10.216 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-28T13:22:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-01-28T13:22:10.323 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-01-28T13:22:10.354 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradospp-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 3/5 2024-01-28T13:22:10.697 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librbd-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 4/5 2024-01-28T13:22:10.837 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : perf-5.14.0-411.el9.x86_64 5/5 2024-01-28T13:22:11.833 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: perf-5.14.0-411.el9.x86_64 5/5 2024-01-28T13:22:11.833 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradospp-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 1/5 2024-01-28T13:22:11.833 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 2/5 2024-01-28T13:22:11.834 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : perf-5.14.0-411.el9.x86_64 3/5 2024-01-28T13:22:11.834 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout: libradospp-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout: librbd-devel-2:19.0.0-933.g00c0d54d.el9.x86_64 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-2.34-7.el9.x86_64 2024-01-28T13:22:12.163 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-01-28T13:22:12.164 INFO:teuthology.orchestra.run.smithi019.stdout: perf-5.14.0-411.el9.x86_64 2024-01-28T13:22:12.164 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:12.164 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-28T13:22:12.543 INFO:tasks.cbt:benchmark: librbdfio 2024-01-28T13:22:12.544 DEBUG:teuthology.orchestra.run.smithi019:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-01-28T13:22:12.558 INFO:teuthology.orchestra.run.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-01-28T13:22:14.717 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-01-28T13:22:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:Operating system Linux 2024-01-28T13:22:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:CPU x86_64 2024-01-28T13:22:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:Big endian no 2024-01-28T13:22:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:Compiler gcc 2024-01-28T13:22:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:Cross compile no 2024-01-28T13:22:15.276 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:22:15.276 INFO:teuthology.orchestra.run.smithi019.stdout:Static build no 2024-01-28T13:22:15.373 INFO:teuthology.orchestra.run.smithi019.stdout:Wordsize 64 2024-01-28T13:22:15.380 INFO:teuthology.orchestra.run.smithi019.stdout:zlib no 2024-01-28T13:22:15.435 INFO:teuthology.orchestra.run.smithi019.stdout:fcntl(F_FULLFSYNC) no 2024-01-28T13:22:15.449 INFO:teuthology.orchestra.run.smithi019.stdout:Linux AIO support no 2024-01-28T13:22:15.449 INFO:teuthology.orchestra.run.smithi019.stdout:Linux AIO support rw flags no 2024-01-28T13:22:15.526 INFO:teuthology.orchestra.run.smithi019.stdout:POSIX AIO support yes 2024-01-28T13:22:15.526 INFO:teuthology.orchestra.run.smithi019.stdout:POSIX AIO support needs -lrt no 2024-01-28T13:22:15.556 INFO:teuthology.orchestra.run.smithi019.stdout:POSIX AIO fsync yes 2024-01-28T13:22:15.585 INFO:teuthology.orchestra.run.smithi019.stdout:POSIX pshared support yes 2024-01-28T13:22:15.635 INFO:teuthology.orchestra.run.smithi019.stdout:pthread_condattr_setclock() yes 2024-01-28T13:22:15.669 INFO:teuthology.orchestra.run.smithi019.stdout:pthread_sigmask() yes 2024-01-28T13:22:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:pthread_getaffinity_np() yes 2024-01-28T13:22:15.713 INFO:teuthology.orchestra.run.smithi019.stdout:Solaris AIO support no 2024-01-28T13:22:15.764 INFO:teuthology.orchestra.run.smithi019.stdout:__sync_fetch_and_add yes 2024-01-28T13:22:15.812 INFO:teuthology.orchestra.run.smithi019.stdout:__sync_synchronize yes 2024-01-28T13:22:15.840 INFO:teuthology.orchestra.run.smithi019.stdout:__sync_val_compare_and_swap yes 2024-01-28T13:22:15.847 INFO:teuthology.orchestra.run.smithi019.stdout:libverbs no 2024-01-28T13:22:15.877 INFO:teuthology.orchestra.run.smithi019.stdout:rdmacm no 2024-01-28T13:22:15.884 INFO:teuthology.orchestra.run.smithi019.stdout:librpma no 2024-01-28T13:22:15.894 INFO:teuthology.orchestra.run.smithi019.stdout:libprotobuf_c no 2024-01-28T13:22:15.925 INFO:teuthology.orchestra.run.smithi019.stdout:asprintf() yes 2024-01-28T13:22:15.954 INFO:teuthology.orchestra.run.smithi019.stdout:vasprintf() yes 2024-01-28T13:22:15.985 INFO:teuthology.orchestra.run.smithi019.stdout:Linux fallocate yes 2024-01-28T13:22:16.017 INFO:teuthology.orchestra.run.smithi019.stdout:POSIX fadvise yes 2024-01-28T13:22:16.048 INFO:teuthology.orchestra.run.smithi019.stdout:POSIX fallocate yes 2024-01-28T13:22:16.077 INFO:teuthology.orchestra.run.smithi019.stdout:sched_setaffinity(3 arg) yes 2024-01-28T13:22:16.077 INFO:teuthology.orchestra.run.smithi019.stdout:sched_setaffinity(2 arg) no 2024-01-28T13:22:16.107 INFO:teuthology.orchestra.run.smithi019.stdout:clock_gettime yes 2024-01-28T13:22:16.137 INFO:teuthology.orchestra.run.smithi019.stdout:CLOCK_MONOTONIC yes 2024-01-28T13:22:16.183 INFO:teuthology.orchestra.run.smithi019.stdout:clockid_t yes 2024-01-28T13:22:16.232 INFO:teuthology.orchestra.run.smithi019.stdout:gettimeofday yes 2024-01-28T13:22:16.264 INFO:teuthology.orchestra.run.smithi019.stdout:fdatasync yes 2024-01-28T13:22:16.294 INFO:teuthology.orchestra.run.smithi019.stdout:pipe() yes 2024-01-28T13:22:16.324 INFO:teuthology.orchestra.run.smithi019.stdout:pipe2() yes 2024-01-28T13:22:16.353 INFO:teuthology.orchestra.run.smithi019.stdout:pread() yes 2024-01-28T13:22:16.438 INFO:teuthology.orchestra.run.smithi019.stdout:sync_file_range yes 2024-01-28T13:22:16.444 INFO:teuthology.orchestra.run.smithi019.stdout:ASharedMemory_create no 2024-01-28T13:22:16.463 INFO:teuthology.orchestra.run.smithi019.stdout:EXT4 move extent yes 2024-01-28T13:22:16.493 INFO:teuthology.orchestra.run.smithi019.stdout:Linux splice(2) yes 2024-01-28T13:22:16.500 INFO:teuthology.orchestra.run.smithi019.stdout:libnuma no 2024-01-28T13:22:16.530 INFO:teuthology.orchestra.run.smithi019.stdout:strsep yes 2024-01-28T13:22:16.559 INFO:teuthology.orchestra.run.smithi019.stdout:strcasestr yes 2024-01-28T13:22:16.585 INFO:teuthology.orchestra.run.smithi019.stdout:strlcat no 2024-01-28T13:22:16.618 INFO:teuthology.orchestra.run.smithi019.stdout:getopt_long_only() yes 2024-01-28T13:22:16.681 INFO:teuthology.orchestra.run.smithi019.stdout:inet_aton yes 2024-01-28T13:22:16.711 INFO:teuthology.orchestra.run.smithi019.stdout:socklen_t yes 2024-01-28T13:22:16.743 INFO:teuthology.orchestra.run.smithi019.stdout:__thread yes 2024-01-28T13:22:16.773 INFO:teuthology.orchestra.run.smithi019.stdout:RUSAGE_THREAD yes 2024-01-28T13:22:16.801 INFO:teuthology.orchestra.run.smithi019.stdout:SCHED_IDLE yes 2024-01-28T13:22:16.834 INFO:teuthology.orchestra.run.smithi019.stdout:TCP_NODELAY yes 2024-01-28T13:22:16.867 INFO:teuthology.orchestra.run.smithi019.stdout:Net engine window_size yes 2024-01-28T13:22:16.901 INFO:teuthology.orchestra.run.smithi019.stdout:TCP_MAXSEG yes 2024-01-28T13:22:16.931 INFO:teuthology.orchestra.run.smithi019.stdout:RLIMIT_MEMLOCK yes 2024-01-28T13:22:16.963 INFO:teuthology.orchestra.run.smithi019.stdout:pwritev/preadv yes 2024-01-28T13:22:16.995 INFO:teuthology.orchestra.run.smithi019.stdout:pwritev2/preadv2 yes 2024-01-28T13:22:17.030 INFO:teuthology.orchestra.run.smithi019.stdout:IPv6 helpers yes 2024-01-28T13:22:17.045 INFO:teuthology.orchestra.run.smithi019.stdout:http engine no 2024-01-28T13:22:17.139 INFO:teuthology.orchestra.run.smithi019.stdout:Rados engine yes 2024-01-28T13:22:17.245 INFO:teuthology.orchestra.run.smithi019.stdout:Rados Block Device engine yes 2024-01-28T13:22:17.356 INFO:teuthology.orchestra.run.smithi019.stdout:rbd_poll yes 2024-01-28T13:22:17.461 INFO:teuthology.orchestra.run.smithi019.stdout:rbd_invalidate_cache yes 2024-01-28T13:22:17.491 INFO:teuthology.orchestra.run.smithi019.stdout:setvbuf yes 2024-01-28T13:22:17.498 INFO:teuthology.orchestra.run.smithi019.stdout:Gluster API engine no 2024-01-28T13:22:17.525 INFO:teuthology.orchestra.run.smithi019.stdout:s390_z196_facilities no 2024-01-28T13:22:17.525 INFO:teuthology.orchestra.run.smithi019.stdout:HDFS engine no 2024-01-28T13:22:17.563 INFO:teuthology.orchestra.run.smithi019.stdout:MTD yes 2024-01-28T13:22:17.570 INFO:teuthology.orchestra.run.smithi019.stdout:libpmem no 2024-01-28T13:22:17.570 INFO:teuthology.orchestra.run.smithi019.stdout:libpmem1_5 no 2024-01-28T13:22:17.577 INFO:teuthology.orchestra.run.smithi019.stdout:libpmem2 no 2024-01-28T13:22:17.577 INFO:teuthology.orchestra.run.smithi019.stdout:PMDK dev-dax engine no 2024-01-28T13:22:17.577 INFO:teuthology.orchestra.run.smithi019.stdout:PMDK libpmem engine no 2024-01-28T13:22:17.584 INFO:teuthology.orchestra.run.smithi019.stdout:DDN's Infinite Memory Engine no 2024-01-28T13:22:17.584 INFO:teuthology.orchestra.run.smithi019.stdout:iscsi engine no 2024-01-28T13:22:17.584 INFO:teuthology.orchestra.run.smithi019.stdout:NBD engine no 2024-01-28T13:22:17.593 INFO:teuthology.orchestra.run.smithi019.stdout:DAOS File System (dfs) Engine no 2024-01-28T13:22:17.601 INFO:teuthology.orchestra.run.smithi019.stdout:NFS engine no 2024-01-28T13:22:17.601 INFO:teuthology.orchestra.run.smithi019.stdout:lex/yacc for arithmetic no 2024-01-28T13:22:17.638 INFO:teuthology.orchestra.run.smithi019.stdout:getmntent yes 2024-01-28T13:22:17.672 INFO:teuthology.orchestra.run.smithi019.stdout:getmntinfo no 2024-01-28T13:22:17.724 INFO:teuthology.orchestra.run.smithi019.stdout:Static Assert yes 2024-01-28T13:22:17.751 INFO:teuthology.orchestra.run.smithi019.stdout:bool yes 2024-01-28T13:22:17.783 INFO:teuthology.orchestra.run.smithi019.stdout:strndup yes 2024-01-28T13:22:17.790 INFO:teuthology.orchestra.run.smithi019.stdout:Valgrind headers no 2024-01-28T13:22:17.821 INFO:teuthology.orchestra.run.smithi019.stdout:Zoned block device support yes 2024-01-28T13:22:17.848 INFO:teuthology.orchestra.run.smithi019.stdout:Zoned block device capacity yes 2024-01-28T13:22:17.855 INFO:teuthology.orchestra.run.smithi019.stdout:libzbc engine no 2024-01-28T13:22:17.889 INFO:teuthology.orchestra.run.smithi019.stdout:NVMe uring command support yes 2024-01-28T13:22:17.903 INFO:teuthology.orchestra.run.smithi019.stdout:xnvme engine no 2024-01-28T13:22:17.910 INFO:teuthology.orchestra.run.smithi019.stdout:isal no 2024-01-28T13:22:17.924 INFO:teuthology.orchestra.run.smithi019.stdout:libblkio engine no 2024-01-28T13:22:17.924 INFO:teuthology.orchestra.run.smithi019.stdout:march_armv8_a_crc_crypto no 2024-01-28T13:22:17.924 INFO:teuthology.orchestra.run.smithi019.stdout:cuda no 2024-01-28T13:22:17.925 INFO:teuthology.orchestra.run.smithi019.stdout:libcufile no 2024-01-28T13:22:17.951 INFO:teuthology.orchestra.run.smithi019.stdout:Build march=native yes 2024-01-28T13:22:17.958 INFO:teuthology.orchestra.run.smithi019.stdout:CUnit no 2024-01-28T13:22:17.986 INFO:teuthology.orchestra.run.smithi019.stdout:__kernel_rwf_t yes 2024-01-28T13:22:18.013 INFO:teuthology.orchestra.run.smithi019.stdout:-Wimplicit-fallthrough=2 yes 2024-01-28T13:22:18.042 INFO:teuthology.orchestra.run.smithi019.stdout:-Wno-stringop-truncation yes 2024-01-28T13:22:18.072 INFO:teuthology.orchestra.run.smithi019.stdout:MADV_HUGEPAGE yes 2024-01-28T13:22:18.101 INFO:teuthology.orchestra.run.smithi019.stdout:gettid yes 2024-01-28T13:22:18.135 INFO:teuthology.orchestra.run.smithi019.stdout:statx(2)/libc yes 2024-01-28T13:22:18.179 INFO:teuthology.orchestra.run.smithi019.stdout:statx(2)/syscall yes 2024-01-28T13:22:18.181 INFO:teuthology.orchestra.run.smithi019.stdout:Windows PDB generation no 2024-01-28T13:22:18.211 INFO:teuthology.orchestra.run.smithi019.stdout:timerfd_create yes 2024-01-28T13:22:18.213 INFO:teuthology.orchestra.run.smithi019.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-01-28T13:22:18.213 INFO:teuthology.orchestra.run.smithi019.stdout:Lib-based ioengines dynamic no 2024-01-28T13:22:18.275 INFO:teuthology.orchestra.run.smithi019.stdout:TCMalloc support yes 2024-01-28T13:22:18.276 INFO:teuthology.orchestra.run.smithi019.stdout:seed_buckets 4 2024-01-28T13:22:18.350 INFO:teuthology.orchestra.run.smithi019.stderr:FIO_VERSION = fio-3.36-53-g625b15 2024-01-28T13:22:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc16.o 2024-01-28T13:22:18.468 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc32.o 2024-01-28T13:22:18.503 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc32c-arm64.o 2024-01-28T13:22:18.585 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc32c-intel.o 2024-01-28T13:22:18.628 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc32c.o 2024-01-28T13:22:18.667 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc64.o 2024-01-28T13:22:18.706 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crc7.o 2024-01-28T13:22:18.738 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/crct10dif_common.o 2024-01-28T13:22:18.773 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/fnv.o 2024-01-28T13:22:18.812 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/md5.o 2024-01-28T13:22:18.926 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/murmur3.o 2024-01-28T13:22:18.967 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/sha1.o 2024-01-28T13:22:19.160 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/sha256.o 2024-01-28T13:22:19.461 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/sha3.o 2024-01-28T13:22:19.610 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/sha512.o 2024-01-28T13:22:19.718 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/test.o 2024-01-28T13:22:19.886 INFO:teuthology.orchestra.run.smithi019.stdout: CC crc/xxhash.o 2024-01-28T13:22:19.982 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/axmap.o 2024-01-28T13:22:20.105 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/bloom.o 2024-01-28T13:22:20.188 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/flist_sort.o 2024-01-28T13:22:20.260 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/gauss.o 2024-01-28T13:22:20.350 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/getrusage.o 2024-01-28T13:22:20.385 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/hweight.o 2024-01-28T13:22:20.422 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/ieee754.o 2024-01-28T13:22:20.470 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/lfsr.o 2024-01-28T13:22:20.572 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/memalign.o 2024-01-28T13:22:20.615 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/memcpy.o 2024-01-28T13:22:20.762 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/mountcheck.o 2024-01-28T13:22:20.801 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/num2str.o 2024-01-28T13:22:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/output_buffer.o 2024-01-28T13:22:20.913 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/pattern.o 2024-01-28T13:22:21.057 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/prio_tree.o 2024-01-28T13:22:21.178 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/rand.o 2024-01-28T13:22:21.370 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/rbtree.o 2024-01-28T13:22:21.495 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/strntol.o 2024-01-28T13:22:21.541 INFO:teuthology.orchestra.run.smithi019.stdout: CC lib/zipf.o 2024-01-28T13:22:21.661 INFO:teuthology.orchestra.run.smithi019.stdout: CC gettime.o 2024-01-28T13:22:21.959 INFO:teuthology.orchestra.run.smithi019.stdout: CC ioengines.o 2024-01-28T13:22:22.234 INFO:teuthology.orchestra.run.smithi019.stdout: CC init.o 2024-01-28T13:22:23.015 INFO:teuthology.orchestra.run.smithi019.stdout: CC stat.o 2024-01-28T13:22:24.923 INFO:teuthology.orchestra.run.smithi019.stdout: CC log.o 2024-01-28T13:22:25.046 INFO:teuthology.orchestra.run.smithi019.stdout: CC time.o 2024-01-28T13:22:25.169 INFO:teuthology.orchestra.run.smithi019.stdout: CC filesetup.o 2024-01-28T13:22:25.881 INFO:teuthology.orchestra.run.smithi019.stdout: CC eta.o 2024-01-28T13:22:26.203 INFO:teuthology.orchestra.run.smithi019.stdout: CC verify.o 2024-01-28T13:22:26.921 INFO:teuthology.orchestra.run.smithi019.stdout: CC memory.o 2024-01-28T13:22:27.091 INFO:teuthology.orchestra.run.smithi019.stdout: CC io_u.o 2024-01-28T13:22:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: CC parse.o 2024-01-28T13:22:28.323 INFO:teuthology.orchestra.run.smithi019.stdout: CC fio_sem.o 2024-01-28T13:22:28.427 INFO:teuthology.orchestra.run.smithi019.stdout: CC rwlock.o 2024-01-28T13:22:28.509 INFO:teuthology.orchestra.run.smithi019.stdout: CC pshared.o 2024-01-28T13:22:28.573 INFO:teuthology.orchestra.run.smithi019.stdout: CC options.o 2024-01-28T13:22:29.383 INFO:teuthology.orchestra.run.smithi019.stdout: CC smalloc.o 2024-01-28T13:22:29.569 INFO:teuthology.orchestra.run.smithi019.stdout: CC filehash.o 2024-01-28T13:22:29.721 INFO:teuthology.orchestra.run.smithi019.stdout: CC profile.o 2024-01-28T13:22:29.840 INFO:teuthology.orchestra.run.smithi019.stdout: CC debug.o 2024-01-28T13:22:29.879 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/cpu.o 2024-01-28T13:22:30.016 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/mmap.o 2024-01-28T13:22:30.182 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/sync.o 2024-01-28T13:22:30.417 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/null.o 2024-01-28T13:22:30.535 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/net.o 2024-01-28T13:22:30.953 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/ftruncate.o 2024-01-28T13:22:31.049 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/fileoperations.o 2024-01-28T13:22:31.190 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/exec.o 2024-01-28T13:22:31.341 INFO:teuthology.orchestra.run.smithi019.stdout: CC server.o 2024-01-28T13:22:32.101 INFO:teuthology.orchestra.run.smithi019.stdout: CC client.o 2024-01-28T13:22:32.655 INFO:teuthology.orchestra.run.smithi019.stdout: CC iolog.o 2024-01-28T13:22:33.108 INFO:teuthology.orchestra.run.smithi019.stdout: CC backend.o 2024-01-28T13:22:33.975 INFO:teuthology.orchestra.run.smithi019.stdout: CC libfio.o 2024-01-28T13:22:34.201 INFO:teuthology.orchestra.run.smithi019.stdout: CC flow.o 2024-01-28T13:22:34.318 INFO:teuthology.orchestra.run.smithi019.stdout: CC cconv.o 2024-01-28T13:22:34.675 INFO:teuthology.orchestra.run.smithi019.stdout: CC gettime-thread.o 2024-01-28T13:22:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: CC helpers.o 2024-01-28T13:22:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: CC json.o 2024-01-28T13:22:35.023 INFO:teuthology.orchestra.run.smithi019.stdout: CC idletime.o 2024-01-28T13:22:35.250 INFO:teuthology.orchestra.run.smithi019.stdout: CC td_error.o 2024-01-28T13:22:35.354 INFO:teuthology.orchestra.run.smithi019.stdout: CC profiles/tiobench.o 2024-01-28T13:22:35.449 INFO:teuthology.orchestra.run.smithi019.stdout: CC profiles/act.o 2024-01-28T13:22:35.610 INFO:teuthology.orchestra.run.smithi019.stdout: CC io_u_queue.o 2024-01-28T13:22:35.659 INFO:teuthology.orchestra.run.smithi019.stdout: CC filelock.o 2024-01-28T13:22:35.785 INFO:teuthology.orchestra.run.smithi019.stdout: CC workqueue.o 2024-01-28T13:22:35.963 INFO:teuthology.orchestra.run.smithi019.stdout: CC rate-submit.o 2024-01-28T13:22:36.111 INFO:teuthology.orchestra.run.smithi019.stdout: CC optgroup.o 2024-01-28T13:22:36.155 INFO:teuthology.orchestra.run.smithi019.stdout: CC helper_thread.o 2024-01-28T13:22:36.338 INFO:teuthology.orchestra.run.smithi019.stdout: CC steadystate.o 2024-01-28T13:22:36.578 INFO:teuthology.orchestra.run.smithi019.stdout: CC zone-dist.o 2024-01-28T13:22:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: CC zbd.o 2024-01-28T13:22:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: CC dedupe.o 2024-01-28T13:22:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: CC fdp.o 2024-01-28T13:22:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/posixaio.o 2024-01-28T13:22:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/falloc.o 2024-01-28T13:22:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/e4defrag.o 2024-01-28T13:22:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/splice.o 2024-01-28T13:22:38.287 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/asprintf.o 2024-01-28T13:22:38.322 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/strlcat.o 2024-01-28T13:22:38.365 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/mtd.o 2024-01-28T13:22:38.526 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/libmtd.o 2024-01-28T13:22:38.911 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/libmtd_legacy.o 2024-01-28T13:22:39.044 INFO:teuthology.orchestra.run.smithi019.stdout: CC diskutil.o 2024-01-28T13:22:39.256 INFO:teuthology.orchestra.run.smithi019.stdout: CC fifo.o 2024-01-28T13:22:39.311 INFO:teuthology.orchestra.run.smithi019.stdout: CC blktrace.o 2024-01-28T13:22:39.650 INFO:teuthology.orchestra.run.smithi019.stdout: CC cgroup.o 2024-01-28T13:22:39.801 INFO:teuthology.orchestra.run.smithi019.stdout: CC trim.o 2024-01-28T13:22:39.918 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/sg.o 2024-01-28T13:22:40.308 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/linux-dev-lookup.o 2024-01-28T13:22:40.363 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/io_uring.o 2024-01-28T13:22:40.750 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/nvme.o 2024-01-28T13:22:41.041 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/linux-blkzoned.o 2024-01-28T13:22:41.194 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/cmdprio.o 2024-01-28T13:22:41.433 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/rados.o 2024-01-28T13:22:41.610 INFO:teuthology.orchestra.run.smithi019.stdout: CC engines/rbd.o 2024-01-28T13:22:41.794 INFO:teuthology.orchestra.run.smithi019.stdout: CC fio.o 2024-01-28T13:22:41.884 INFO:teuthology.orchestra.run.smithi019.stdout: LINK fio 2024-01-28T13:22:42.080 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/log.o 2024-01-28T13:22:42.120 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/genzipf.o 2024-01-28T13:22:42.260 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/strcasestr.o 2024-01-28T13:22:42.283 INFO:teuthology.orchestra.run.smithi019.stdout: CC oslib/strndup.o 2024-01-28T13:22:42.306 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/fio-genzipf 2024-01-28T13:22:42.397 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/btrace2fio.o 2024-01-28T13:22:42.825 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/fio-btrace2fio 2024-01-28T13:22:42.915 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/verify-state.o 2024-01-28T13:22:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/debug.o 2024-01-28T13:22:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/fio-verify-state 2024-01-28T13:22:43.136 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/stest.o 2024-01-28T13:22:43.204 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/arch.o 2024-01-28T13:22:43.233 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/stest 2024-01-28T13:22:43.330 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/ieee754.o 2024-01-28T13:22:43.368 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/ieee754 2024-01-28T13:22:43.457 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/axmap.o 2024-01-28T13:22:43.557 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/axmap 2024-01-28T13:22:43.648 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/lfsr-test.o 2024-01-28T13:22:43.742 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/lfsr-test 2024-01-28T13:22:43.832 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/gen-rand.o 2024-01-28T13:22:43.929 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/gen-rand 2024-01-28T13:22:44.020 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/memlock.o 2024-01-28T13:22:44.079 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/memlock 2024-01-28T13:22:44.167 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/read-to-pipe-async.o 2024-01-28T13:22:44.356 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/read-to-pipe-async 2024-01-28T13:22:44.444 INFO:teuthology.orchestra.run.smithi019.stdout: CC t/io_uring.o 2024-01-28T13:22:44.991 INFO:teuthology.orchestra.run.smithi019.stdout: LINK t/io_uring 2024-01-28T13:22:45.082 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-01-28T13:22:45.109 DEBUG:teuthology.orchestra.run.smithi019:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-28T13:22:45.353 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - Settings.cluster: 2024-01-28T13:22:45.353 INFO:teuthology.orchestra.run.smithi019.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-01-28T13:22:45.353 INFO:teuthology.orchestra.run.smithi019.stderr: 'clients': ['ubuntu@smithi019.front.sepia.ceph.com'], 2024-01-28T13:22:45.353 INFO:teuthology.orchestra.run.smithi019.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-01-28T13:22:45.353 INFO:teuthology.orchestra.run.smithi019.stderr: 'head': 'ubuntu@smithi019.front.sepia.ceph.com', 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'is_teuthology': True, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'iterations': 1, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'mons': ['ubuntu@smithi019.front.sepia.ceph.com'], 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'osds': ['ubuntu@smithi019.front.sepia.ceph.com'], 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'osds_per_node': 3, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'pid_dir': '/var/run/ceph', 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'rebuild_every_test': False, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'rgws': {'ubuntu@smithi019.front.sepia.ceph.com': None}, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'tmp_dir': '/tmp/cbt', 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'use_existing': True, 2024-01-28T13:22:45.354 INFO:teuthology.orchestra.run.smithi019.stderr: 'user': 'ubuntu'} 2024-01-28T13:22:45.355 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.355 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.355 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.355 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite; fi 2024-01-28T13:22:45.357 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.357 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.357 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2024-01-28T13:22:45.360 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-01-28T13:22:45.360 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.360 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.360 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl 2024-01-28T13:22:45.362 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.362 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-28T13:22:45.363 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.364 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.364 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/perf 2024-01-28T13:22:45.366 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:22:45.366 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-28T13:22:45.366 INFO:teuthology.orchestra.run.smithi019.stderr:13:22:45 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-28T13:23:45.417 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.417 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.417 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-28T13:23:45.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-28T13:23:45.487 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.487 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.487 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* 2024-01-28T13:23:45.504 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.504 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/$(basename ${i}).smithi019.front.sepia.ceph.com ; done 2024-01-28T13:23:45.509 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.509 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.509 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl 2024-01-28T13:23:45.511 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.511 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-28T13:23:45.513 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.513 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.513 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/perf 2024-01-28T13:23:45.516 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.516 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-28T13:23:45.516 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-28T13:23:45.517 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.517 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.517 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-01-28T13:23:45.820 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.820 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:45.820 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-01-28T13:23:46.711 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:46 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:46.711 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:46 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:46.711 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-01-28T13:23:47.710 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:47 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:47.710 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:47 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:47.710 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-01-28T13:23:48.711 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:48 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:48.711 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:48 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:48.711 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-01-28T13:23:49.713 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:49 - INFO - cbt - Checking Health after pool creation. 2024-01-28T13:23:54.719 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:54 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:23:54.719 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:54 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:23:54.719 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-28T13:23:55.048 INFO:teuthology.orchestra.run.smithi019.stderr:13:23:55 - INFO - cbt - Final Pool Health Check. 2024-01-28T13:24:00.054 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:00.054 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:00.054 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-28T13:24:00.367 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:00.368 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:00.368 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-01-28T13:24:00.370 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:00.370 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:00.370 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi019.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-01-28T13:24:02.137 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.137 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.137 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi019.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-01-28T13:24:02.235 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.235 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.235 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-28T13:24:02.251 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.251 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.252 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-28T13:24:02.281 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - INFO - cbt - Attempting to populating fio files... 2024-01-28T13:24:02.281 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.281 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.281 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-28T13:24:02.283 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.283 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:24:02.283 INFO:teuthology.orchestra.run.smithi019.stderr:13:24:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-28T13:26:37.449 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.449 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd. 2024-01-28T13:26:37.449 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - INFO - cbt - Setting OSD Read Ahead to: 4096 2024-01-28T13:26:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2024-01-28T13:26:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2024-01-28T13:26:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.451 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.451 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2024-01-28T13:26:37.466 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.466 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.466 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-01-28T13:26:37.779 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:26:37.779 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-01-28T13:26:38.538 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:38 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:26:38.538 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:38 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:26:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2024-01-28T13:26:38.665 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:38 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:26:38.666 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:38 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:26:38.666 INFO:teuthology.orchestra.run.smithi019.stderr:13:26:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/ceph_settings.out 2024-01-28T13:27:04.580 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.580 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.581 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pgautoscaler.log 2024-01-28T13:27:04.932 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.933 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.933 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/collectl 2024-01-28T13:27:04.935 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.935 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-28T13:27:04.936 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.937 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.937 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf 2024-01-28T13:27:04.939 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.939 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-28T13:27:04.940 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-28T13:27:04.940 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - INFO - cbt - Running rbd fio randwrite test. 2024-01-28T13:27:04.940 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2024-01-28T13:27:04.940 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.940 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.940 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 2024-01-28T13:27:04.941 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2024-01-28T13:27:04.941 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.941 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:27:04.941 INFO:teuthology.orchestra.run.smithi019.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 2024-01-28T13:28:05.960 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:05 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:05.961 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:05 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:05.961 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-28T13:28:05.982 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:05.983 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:05.983 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-28T13:28:06.012 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.012 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf/perf.data 2024-01-28T13:28:06.013 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.013 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf/perf_stat.* 2024-01-28T13:28:06.015 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.015 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.015 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/historic_ops.out 2024-01-28T13:28:06.330 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.594 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.594 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* 2024-01-28T13:28:06.612 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.612 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/$(basename ${i}).smithi019.front.sepia.ceph.com ; done 2024-01-28T13:28:06.990 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:07 - INFO - cbt - Convert results to json format. 2024-01-28T13:28:06.990 INFO:teuthology.orchestra.run.smithi019.stderr:13:28:07 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.991 INFO:teuthology.orchestra.run.smithi019.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2024-01-28T13:28:06.991 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com: smithi019.front.sepia.ceph.com 2024-01-28T13:28:06.991 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-28T13:28:06.991 INFO:teuthology.orchestra.run.smithi019.stdout:['smithi019.front.sepia.ceph.com'] 2024-01-28T13:28:07.100 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-01-28T13:28:07.112 DEBUG:teuthology.orchestra.run.smithi019:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-01-28T13:28:07.227 DEBUG:teuthology.orchestra.run.smithi019:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-01-28T13:28:07.304 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-01-28T13:28:07.305 INFO:tasks.cbt_performance:reading cbt preferences 2024-01-28T13:28:07.305 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-01-28T13:28:07.360 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.0.smithi019.front.sepia.ceph.com 2024-01-28T13:28:07.361 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.1.smithi019.front.sepia.ceph.com 2024-01-28T13:28:07.361 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:28:07.362 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.0.smithi019.front.sepia.ceph.com of=/dev/stdout 2024-01-28T13:28:07.382 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-28T13:28:07.382 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd -name 'perf_stat.*' 2024-01-28T13:28:07.437 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-28T13:28:07.437 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-28T13:28:07.437 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-28T13:28:07.438 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.1.smithi019.front.sepia.ceph.com of=/dev/stdout 2024-01-28T13:28:07.493 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-28T13:28:07.494 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd -name 'perf_stat.*' 2024-01-28T13:28:07.548 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-28T13:28:07.549 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-28T13:28:07.560 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-28T13:28:07.568 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-28T13:28:07.569 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-28T13:28:07.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-28T13:28:07.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:28:07.711+0000 7f411879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:28:07.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-28T13:28:07.719+0000 7f411879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-28T13:28:07.920 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-28T13:28:07.921 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-28T13:28:07.921 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-28T13:28:07.921 INFO:teuthology.orchestra.run:waiting for 300 2024-01-28T13:28:12.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-28T13:28:14.022 INFO:tasks.ceph.osd.0:Stopped 2024-01-28T13:28:14.023 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-28T13:28:14.023 INFO:teuthology.orchestra.run:waiting for 300 2024-01-28T13:28:17.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-28T13:28:17.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-28T13:28:20.124 INFO:tasks.ceph.osd.1:Stopped 2024-01-28T13:28:20.124 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-28T13:28:20.124 INFO:teuthology.orchestra.run:waiting for 300 2024-01-28T13:28:22.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-28T13:28:22.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-28T13:28:26.226 INFO:tasks.ceph.osd.2:Stopped 2024-01-28T13:28:26.226 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-28T13:28:26.227 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-28T13:28:26.227 INFO:teuthology.orchestra.run:waiting for 300 2024-01-28T13:28:26.259 INFO:tasks.ceph.mgr.x:Stopped 2024-01-28T13:28:26.260 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-28T13:28:26.260 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-28T13:28:26.260 INFO:teuthology.orchestra.run:waiting for 300 2024-01-28T13:28:26.312 INFO:tasks.ceph.mon.a:Stopped 2024-01-28T13:28:26.312 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-28T13:28:26.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-28T13:28:26.418 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:26.418 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-28T13:28:26.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:26.641 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-28T13:28:26.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:26.841 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-28T13:28:26.897 INFO:tasks.ceph:Archiving mon data... 2024-01-28T13:28:26.899 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830/data/mon.a.tgz 2024-01-28T13:28:26.899 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-01-28T13:28:26.921 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.12ki7j2crE 2024-01-28T13:28:26.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.12ki7j2crE 2024-01-28T13:28:27.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.12ki7j2crE 2024-01-28T13:28:27.147 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.12ki7j2crE is 205KB 2024-01-28T13:28:27.240 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.12ki7j2crE 2024-01-28T13:28:27.254 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-28T13:28:27.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-28T13:28:27.379 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-28T13:28:27.379 INFO:tasks.ceph:Archiving crash dumps... 2024-01-28T13:28:27.381 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830/remote/smithi019/crash 2024-01-28T13:28:27.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-28T13:28:27.445 INFO:tasks.ceph:Compressing logs... 2024-01-28T13:28:27.445 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-28T13:28:27.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29241.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27794.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29241.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.27794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29153.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29241.log.gz 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30605.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29351.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30605.log: -- /var/log/ceph/ceph-client.admin.34092.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30476.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30351.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34352.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30185.log 2024-01-28T13:28:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29589.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30269.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29589.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29153.log.gz 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30563.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34303.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34092.log.gz 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30563.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30647.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34320.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29617.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30647.log: /var/log/ceph/ceph-client.admin.34320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30082.log 2024-01-28T13:28:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29431.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.30082.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30124.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34256.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30124.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27794.log.gz 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30026.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34208.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.30026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34233.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34208.log: /var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34765.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-28T13:28:27.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34233.log: /var/log/ceph/ceph-client.admin.34765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34351.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34765.log.gz 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30227.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34183.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29589.log.gz 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30227.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30477.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34183.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34764.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29617.log.gz 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30349.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34764.log.gz 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34717.log 2024-01-28T13:28:27.532 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34256.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34279.log 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34156.log 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34717.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30437.log 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29391.log 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30350.log 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.29391.log: /var/log/ceph/ceph-client.admin.30350.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30026.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30082.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30124.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30185.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30227.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30269.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34156.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30350.log.gz 2024-01-28T13:28:27.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30349.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30351.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30437.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30477.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30476.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30563.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30605.log.gz 2024-01-28T13:28:27.534 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30647.log.gz 2024-01-28T13:28:27.535 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34279.log.gz 2024-01-28T13:28:27.535 INFO:teuthology.orchestra.run.smithi019.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.34303.log.gz 2024-01-28T13:28:27.535 INFO:teuthology.orchestra.run.smithi019.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.34320.log.gz 2024-01-28T13:28:27.535 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34351.log.gz 2024-01-28T13:28:27.535 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34352.log.gz 2024-01-28T13:28:27.535 INFO:teuthology.orchestra.run.smithi019.stderr: 85.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-28T13:28:27.537 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34208.log.gz 2024-01-28T13:28:27.538 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34233.log.gz 2024-01-28T13:28:27.541 INFO:teuthology.orchestra.run.smithi019.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-28T13:28:27.563 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34183.log.gz 2024-01-28T13:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29351.log.gz 2024-01-28T13:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29391.log.gz 2024-01-28T13:28:27.606 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29431.log.gz 2024-01-28T13:28:27.677 INFO:teuthology.orchestra.run.smithi019.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-28T13:28:27.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-28T13:28:45.050 INFO:teuthology.orchestra.run.smithi019.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-28T13:28:45.123 INFO:teuthology.orchestra.run.smithi019.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-28T13:28:45.719 INFO:teuthology.orchestra.run.smithi019.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-28T13:28:45.721 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-01-28T13:28:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m18.222s 2024-01-28T13:28:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:user 0m11.828s 2024-01-28T13:28:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m0.597s 2024-01-28T13:28:45.721 INFO:tasks.ceph:Archiving logs... 2024-01-28T13:28:45.722 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830/remote/smithi019/log 2024-01-28T13:28:45.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-28T13:28:47.588 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-28T13:28:47.597 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-28T13:28:47.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-28T13:28:47.656 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-28T13:28:47.665 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-28T13:28:47.665 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-28T13:28:47.687 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-01-28T13:28:47.748 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-28T13:28:47.748 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-01-28T13:28:47.748 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +687us[ +682us] +/- 84ms 2024-01-28T13:28:47.748 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -791us[ -795us] +/- 67ms 2024-01-28T13:28:47.748 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +248us[ +248us] +/- 89ms 2024-01-28T13:28:47.748 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-28T13:28:47.766 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-28T13:28:47.782 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-28T13:28:47.783 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-28T13:28:47.847 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-28T13:28:47.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-28T13:28:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-28T13:28:47.997 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-28T13:28:48.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-28T13:28:48.149 INFO:teuthology.task.internal:Duration was 890.974492 seconds 2024-01-28T13:28:48.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-28T13:28:48.203 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-28T13:28:48.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-28T13:28:48.295 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-28T13:28:48.701 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-28T13:28:48.701 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-01-28T13:28:48.701 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-28T13:28:48.723 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-28T13:28:48.723 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-28T13:28:48.794 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-28T13:28:48.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-28T13:28:48.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-28T13:28:48.955 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-28T13:28:48.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-28T13:28:48.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-28T13:28:49.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-28T13:28:49.075 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-01-28T13:28:49.098 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-28T13:28:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-28T13:28:49.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-28T13:28:49.163 INFO:teuthology.task.internal:Transferring archived files... 2024-01-28T13:28:49.164 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-01-28_12:38:29-crimson-rados-wip-matanb-crimson-pg-map-logs-distro-crimson-smithi/7535830/remote/smithi019 2024-01-28T13:28:49.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-28T13:28:49.259 INFO:teuthology.task.internal:Removing archive directory... 2024-01-28T13:28:49.260 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-28T13:28:49.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-28T13:28:49.313 INFO:teuthology.task.internal:Not uploading archives. 2024-01-28T13:28:49.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-28T13:28:49.323 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-28T13:28:49.323 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-28T13:28:49.356 INFO:teuthology.orchestra.run.smithi019.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 28 13:28 /home/ubuntu/cephtest 2024-01-28T13:28:49.357 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-28T13:28:49.368 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-28T13:28:49.390 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-matanb-crimson-pg-map-logs&var-job_id_selected=7535830 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} duration: 890.9744915962219 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-01-28T13:28:49.390 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-28T13:28:49.485 INFO:teuthology.run:pass