2023-11-08T11:57:35.017 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T11:57:35.018 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916 --name matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi --owner scheduled_matan@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} -- /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916/orig.config.yaml 2023-11-08T11:57:35.062 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T11:57:35.144 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916 branch: wip-matanb-crimson-do_osd_ops_execute-v3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} email: null first_in_suite: false job_id: '7451916' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-do_osd_ops_execute-v3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: db6fbc96537be451d7eb9f6302778fac84588cb6 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6890 sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: db6fbc96537be451d7eb9f6302778fac84588cb6 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMPwXdIR+bESh0S4xXdY2pTg+iFop4Ey4+AAMLzCrhE5j9QdSGdCam+TJ2R+WU2BKgSobadmnlK8/EEWU3mI4zg= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-08_11:46:17 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 2023-11-08T11:57:35.145 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa; will attempt to use it 2023-11-08T11:57:35.146 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa/tasks 2023-11-08T11:57:35.146 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-08T11:57:35.149 INFO:teuthology.task.internal:Checking packages... 2023-11-08T11:57:35.174 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '25e18238395a1cded51f624e1e84ec38cbcb1b4a' 2023-11-08T11:57:35.174 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-08T11:57:35.175 INFO:teuthology.packaging:ref: None 2023-11-08T11:57:35.175 INFO:teuthology.packaging:tag: None 2023-11-08T11:57:35.175 INFO:teuthology.packaging:branch: wip-matanb-crimson-do_osd_ops_execute-v3 2023-11-08T11:57:35.176 INFO:teuthology.packaging:sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a 2023-11-08T11:57:35.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-do_osd_ops_execute-v3 2023-11-08T11:57:35.328 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7227.g25e18238 2023-11-08T11:57:35.329 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-08T11:57:35.336 INFO:teuthology.task.internal:no buildpackages task found 2023-11-08T11:57:35.336 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-08T11:57:35.356 INFO:teuthology.task.internal:Saving configuration 2023-11-08T11:57:35.375 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-08T11:57:35.380 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T11:57:35.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 11:50:44.119604', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMPwXdIR+bESh0S4xXdY2pTg+iFop4Ey4+AAMLzCrhE5j9QdSGdCam+TJ2R+WU2BKgSobadmnlK8/EEWU3mI4zg='} 2023-11-08T11:57:35.407 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-08T11:57:35.412 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-08T11:57:35.412 INFO:teuthology.run_tasks:Running task console_log... 2023-11-08T11:57:35.459 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdb997de940>, signals=[15]) 2023-11-08T11:57:35.460 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-08T11:57:35.465 INFO:teuthology.task.internal:Opening connections... 2023-11-08T11:57:35.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T11:57:35.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T11:57:35.566 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-08T11:57:35.572 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2023-11-08T11:57:35.635 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2023-11-08T11:57:35.636 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2023-11-08T11:57:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2023-11-08T11:57:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2023-11-08T11:57:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2023-11-08T11:57:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2023-11-08T11:57:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2023-11-08T11:57:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2023-11-08T11:57:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-08T11:57:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2023-11-08T11:57:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2023-11-08T11:57:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-08T11:57:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2023-11-08T11:57:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T11:57:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-08T11:57:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T11:57:35.695 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2023-11-08T11:57:35.716 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-08T11:57:35.725 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-08T11:57:35.847 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-08T11:57:35.847 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2023-11-08T11:57:35.865 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-08T11:57:35.884 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-08T11:57:35.884 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2023-11-08T11:57:35.929 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-08T11:57:35.965 INFO:teuthology.run_tasks:Running task kernel... 2023-11-08T11:57:35.982 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-08T11:57:35.983 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-08T11:57:35.983 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-08T11:57:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T11:57:36.000 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-11-08T11:57:36.057 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-361.el9.x86_64 2023-11-08T11:57:36.058 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-361.el9.x86_64 2023-11-08T11:57:36.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2023-11-08T11:57:38.047 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-11-08T11:57:39.406 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-11-08T11:57:40.330 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-11-08T11:57:56.571 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 18 MB 00:15 2023-11-08T11:58:01.196 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 5.8 MB 00:01 2023-11-08T11:58:02.789 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-11-08T11:58:04.240 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 15 MB/s | 19 MB 00:01 2023-11-08T11:58:09.115 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-08T11:58:11.488 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-08T11:58:11.489 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-08T11:58:11.586 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T11:58:11.595 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T11:58:11.595 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T11:58:11.595 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T11:58:11.596 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T11:58:11.596 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-382.el9 CentOS-BaseOS 5.4 M 2023-11-08T11:58:11.596 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T11:58:11.596 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-382.el9 CentOS-BaseOS 20 M 2023-11-08T11:58:11.597 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-382.el9 CentOS-BaseOS 38 M 2023-11-08T11:58:11.597 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-382.el9 CentOS-BaseOS 32 M 2023-11-08T11:58:11.597 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T11:58:11.597 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T11:58:11.597 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T11:58:11.597 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2023-11-08T11:58:11.598 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T11:58:11.598 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 95 M 2023-11-08T11:58:11.599 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 121 M 2023-11-08T11:58:11.599 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T11:58:11.925 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-382.el9.x86_64.rpm 17 MB/s | 5.4 MB 00:00 2023-11-08T11:58:12.184 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-382.el9.x86_64.rpm 34 MB/s | 20 MB 00:00 2023-11-08T11:58:12.576 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-core-5.14.0-382.el9.x86_6 50 MB/s | 32 MB 00:00 2023-11-08T11:58:13.059 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-5.14.0-382.el9.x86_64.rpm 26 MB/s | 38 MB 00:01 2023-11-08T11:58:13.062 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T11:58:13.062 INFO:teuthology.orchestra.run.smithi049.stdout:Total 65 MB/s | 95 MB 00:01 2023-11-08T11:58:13.063 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T11:58:13.491 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T11:58:13.492 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T11:58:14.544 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T11:58:14.545 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T11:58:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T11:58:17.688 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-382.el9.x86_64 1/4 2023-11-08T11:58:17.818 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-382.el9.x86_64 2/4 2023-11-08T11:58:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-382.el9.x86_64 2/4 2023-11-08T11:58:20.867 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-382.el9.x86_64 3/4 2023-11-08T11:58:25.807 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-382.el9.x86_64 3/4 2023-11-08T11:58:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-382.el9.x86_64 4/4 2023-11-08T11:58:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-382.el9.x86_64 4/4 2023-11-08T11:59:01.467 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-382.el9.x86_64 4/4 2023-11-08T11:59:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-382.el9.x86_64 4/4 2023-11-08T11:59:01.961 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-382.el9.x86_64 4/4 2023-11-08T11:59:01.962 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-382.el9.x86_64 1/4 2023-11-08T11:59:01.962 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-382.el9.x86_64 2/4 2023-11-08T11:59:01.962 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-382.el9.x86_64 3/4 2023-11-08T11:59:02.401 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-382.el9.x86_64 4/4 2023-11-08T11:59:02.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T11:59:02.402 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T11:59:02.402 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-382.el9.x86_64 2023-11-08T11:59:02.402 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-382.el9.x86_64 2023-11-08T11:59:02.402 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-382.el9.x86_64 2023-11-08T11:59:02.402 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-382.el9.x86_64 2023-11-08T11:59:02.403 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T11:59:02.403 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T11:59:02.934 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T11:59:03.083 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-382.el9.x86_64 2023-11-08T11:59:03.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-382.el9.x86_64 2023-11-08T11:59:03.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-382.el9.x86_64 2023-11-08T11:59:03.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-08T11:59:03.084 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-08T11:59:03.085 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-08T11:59:03.085 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2023-11-08T11:59:03.085 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T11:59:03.195 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-382.el9.x86_64 2023-11-08T11:59:03.195 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-382.el9.x86_64 2023-11-08T11:59:03.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2023-11-08T11:59:03.238 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2023-11-08T11:59:03.239 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2023-11-08T11:59:03.239 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2023-11-08T11:59:03.239 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 68.el9 2023-11-08T11:59:03.239 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2023-11-08T11:59:03.240 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-08T11:59:03.240 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2023-11-08T11:59:03.240 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8264254 2023-11-08T11:59:03.240 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2023-11-08T11:59:03.240 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-08T11:59:03.241 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-08T11:59:03.241 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-08T11:59:03.241 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-08T11:59:03.241 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2023-11-08T11:59:03.241 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2023-11-08T11:59:03.242 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-08T11:59:03.242 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2023-11-08T11:59:03.242 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2023-11-08T11:59:03.242 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T11:59:03.242 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-08T11:59:03.243 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-08T11:59:03.243 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-08T11:59:03.243 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2023-11-08T11:59:03.243 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T11:59:03.243 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2023-11-08T11:59:03.244 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-08T11:59:03.245 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-382.el9.x86_64 2023-11-08T11:59:03.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-08T11:59:03.934 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2023-11-08T11:59:05.288 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-08T11:59:05.323 INFO:teuthology.orchestra.run.smithi049.stderr:done 2023-11-08T11:59:05.325 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-11-08T11:59:05.341 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.Pk58vz1w6e 2023-11-08T11:59:05.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Pk58vz1w6e 2023-11-08T11:59:05.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.Pk58vz1w6e 2023-11-08T11:59:05.552 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.Pk58vz1w6e is 6KB 2023-11-08T11:59:05.601 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.Pk58vz1w6e 2023-11-08T11:59:05.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2023-11-08T11:59:05.685 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-08T11:59:05.685 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-08T11:59:05.686 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-08T11:59:05.686 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-382.el9.x86_64.conf 2023-11-08T11:59:05.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-382.el9.x86_64 2023-11-08T11:59:05.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2023-11-08T11:59:35.870 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-08T11:59:35.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T11:59:35.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T11:59:54.235 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-08T12:00:03.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:00:03.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:00:06.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-08T12:00:18.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:00:18.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:00:21.371 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-08T12:00:36.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:00:36.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:00:39.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-08T12:00:57.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:00:57.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:01:57.527 DEBUG:teuthology.orchestra.remote:timed out 2023-11-08T12:02:18.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:02:18.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:02:18.914 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-11-08T12:02:19.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:02:19.318 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-382.el9.x86_64"... 2023-11-08T12:02:19.318 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-11-08T12:02:19.333 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-382.el9.x86_64 2023-11-08T12:02:19.334 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-382.el9.x86_64 vs 5.14.0-382.el9.x86_64 2023-11-08T12:02:19.334 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-08T12:02:19.335 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-08T12:02:20.335 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-08T12:02:20.336 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-08T12:02:20.420 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2023-11-08T12:02:20.456 DEBUG:teuthology.parallel:result is None 2023-11-08T12:02:20.456 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-08T12:02:20.464 INFO:teuthology.task.internal:Creating test directory... 2023-11-08T12:02:20.464 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-08T12:02:20.483 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-08T12:02:20.490 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-08T12:02:20.517 INFO:teuthology.task.internal:Creating archive directory... 2023-11-08T12:02:20.518 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-08T12:02:20.552 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-08T12:02:20.560 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-08T12:02:20.561 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-08T12:02:20.630 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T12:02:20.647 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T12:02:20.649 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-08T12:02:20.656 INFO:teuthology.task.internal:Configuring sudo... 2023-11-08T12:02:20.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-08T12:02:20.689 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-08T12:02:20.698 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-08T12:02:20.699 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-08T12:02:20.745 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T12:02:20.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T12:02:20.878 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T12:02:20.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T12:02:21.006 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:02:21.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-08T12:02:21.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2023-11-08T12:02:21.163 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T12:02:21.336 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-08T12:02:21.344 INFO:teuthology.task.internal:Starting timer... 2023-11-08T12:02:21.344 INFO:teuthology.run_tasks:Running task pcp... 2023-11-08T12:02:21.354 INFO:teuthology.run_tasks:Running task selinux... 2023-11-08T12:02:21.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2023-11-08T12:02:21.437 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2023-11-08T12:02:21.439 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-08T12:02:21.439 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2023-11-08T12:02:21.463 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2023-11-08T12:02:21.463 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive'} 2023-11-08T12:02:21.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T12:02:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:02:21.535 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-08T12:02:21.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2023-11-08T12:02:21.615 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-08T12:02:21.625 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-08T12:02:21.626 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-08T12:02:21.643 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-08T12:02:21.645 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-08T12:07:56.972 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2023-11-08T12:07:56.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:07:56.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:07:57.072 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-11-08T12:07:57.141 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-08T12:07:57.141 INFO:teuthology.run_tasks:Running task clock... 2023-11-08T12:07:57.154 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-08T12:07:57.154 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-08T12:07:57.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T12:07:57.216 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-08T12:07:57.236 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-08T12:07:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2023-11-08T12:07:57.292 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2023-11-08T12:07:57.314 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-08T12:07:57.335 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-08T12:07:57.385 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2023-11-08T12:07:57.388 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T12:07:57.389 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-11-08T12:07:57.389 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:07:57.389 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:07:57.389 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:07:57.389 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:07:57.391 INFO:teuthology.run_tasks:Running task install... 2023-11-08T12:07:57.399 DEBUG:teuthology.task.install:project ceph 2023-11-08T12:07:57.400 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'}} 2023-11-08T12:07:57.400 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'} 2023-11-08T12:07:57.400 INFO:teuthology.task.install:Using flavor: crimson 2023-11-08T12:07:57.406 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-08T12:07:57.407 INFO:teuthology.task.install:extra packages: [] 2023-11-08T12:07:57.407 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a', 'tag': None, 'wait_for_package': False} 2023-11-08T12:07:57.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=25e18238395a1cded51f624e1e84ec38cbcb1b4a 2023-11-08T12:07:57.703 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/ 2023-11-08T12:07:57.704 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7227.g25e18238 2023-11-08T12:07:57.892 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-08T12:07:57.893 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:07:57.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-08T12:07:57.929 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-08T12:07:57.930 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/25e18238395a1cded51f624e1e84ec38cbcb1b4a/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-08T12:07:58.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-08T12:07:58.090 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-08T12:07:58.119 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2023-11-08T12:07:58.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2023-11-08T12:07:58.367 INFO:teuthology.orchestra.run.smithi049.stdout:72 files removed 2023-11-08T12:07:58.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2023-11-08T12:07:59.143 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 197 kB/s | 81 kB 00:00 2023-11-08T12:07:59.569 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-11-08T12:07:59.969 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2023-11-08T12:08:00.346 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-11-08T12:08:01.697 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-11-08T12:08:02.529 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-11-08T12:08:07.030 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 18 MB 00:03 2023-11-08T12:08:11.706 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 5.8 MB 00:01 2023-11-08T12:08:13.098 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2023-11-08T12:08:15.782 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 19 MB 00:02 2023-11-08T12:08:20.541 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2023-11-08T12:08:23.269 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:08:23.276 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:08:23.277 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:08:23.277 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:08:23.277 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:08:23.277 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:18.0.0-7227.g25e18238.el9 ceph 8.4 M 2023-11-08T12:08:23.277 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2023-11-08T12:08:23.278 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 3.1 M 2023-11-08T12:08:23.278 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 3.0 M 2023-11-08T12:08:23.278 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:08:23.278 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options 2023-11-08T12:08:23.278 INFO:teuthology.orchestra.run.smithi049.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-08T12:08:23.279 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:18.0.0-7227.g25e18238.el9 ceph 6.6 M 2023-11-08T12:08:23.279 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:18.0.0-7227.g25e18238.el9 ceph 17 M 2023-11-08T12:08:23.279 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:18.0.0-7227.g25e18238.el9 ceph 24 k 2023-11-08T12:08:23.279 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-08T12:08:23.280 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-08T12:08:23.280 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-08T12:08:23.280 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 728 k 2023-11-08T12:08:23.280 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-08T12:08:23.280 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-08T12:08:23.281 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-08T12:08:23.281 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 470 k 2023-11-08T12:08:23.281 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-08T12:08:23.281 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 4.5 M 2023-11-08T12:08:23.281 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-08T12:08:23.281 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-08T12:08:23.282 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-08T12:08:23.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse 2023-11-08T12:08:23.282 INFO:teuthology.orchestra.run.smithi049.stdout: x86_64 2:18.0.0-7227.g25e18238.el9 ceph 45 k 2023-11-08T12:08:23.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:18.0.0-7227.g25e18238.el9 ceph 127 k 2023-11-08T12:08:23.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:18.0.0-7227.g25e18238.el9 ceph 162 k 2023-11-08T12:08:23.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-08T12:08:23.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:18.0.0-7227.g25e18238.el9 ceph 320 k 2023-11-08T12:08:23.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:18.0.0-7227.g25e18238.el9 ceph 298 k 2023-11-08T12:08:23.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:18.0.0-7227.g25e18238.el9 ceph 99 k 2023-11-08T12:08:23.283 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-08T12:08:23.284 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-08T12:08:23.284 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:23.284 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:08:23.284 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:08:23.284 INFO:teuthology.orchestra.run.smithi049.stdout:Install 27 Packages 2023-11-08T12:08:23.285 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2023-11-08T12:08:23.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:23.285 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 54 M 2023-11-08T12:08:23.286 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:08:24.451 INFO:teuthology.orchestra.run.smithi049.stdout:(1/29): ceph-base-18.0.0-7227.g25e18238.el9.x86 7.9 MB/s | 6.6 MB 00:00 2023-11-08T12:08:24.526 INFO:teuthology.orchestra.run.smithi049.stdout:(2/29): ceph-selinux-18.0.0-7227.g25e18238.el9. 322 kB/s | 24 kB 00:00 2023-11-08T12:08:24.634 INFO:teuthology.orchestra.run.smithi049.stdout:(3/29): libcephfs2-18.0.0-7227.g25e18238.el9.x8 6.6 MB/s | 728 kB 00:00 2023-11-08T12:08:24.735 INFO:teuthology.orchestra.run.smithi049.stdout:(4/29): libradosstriper1-18.0.0-7227.g25e18238. 4.6 MB/s | 470 kB 00:00 2023-11-08T12:08:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:(5/29): librgw2-18.0.0-7227.g25e18238.el9.x86_6 12 MB/s | 4.5 MB 00:00 2023-11-08T12:08:25.177 INFO:teuthology.orchestra.run.smithi049.stdout:(6/29): python3-ceph-argparse-18.0.0-7227.g25e1 780 kB/s | 45 kB 00:00 2023-11-08T12:08:25.277 INFO:teuthology.orchestra.run.smithi049.stdout:(7/29): python3-ceph-common-18.0.0-7227.g25e182 1.2 MB/s | 127 kB 00:00 2023-11-08T12:08:25.394 INFO:teuthology.orchestra.run.smithi049.stdout:(8/29): python3-cephfs-18.0.0-7227.g25e18238.el 1.4 MB/s | 162 kB 00:00 2023-11-08T12:08:25.511 INFO:teuthology.orchestra.run.smithi049.stdout:(9/29): ceph-radosgw-18.0.0-7227.g25e18238.el9. 4.4 MB/s | 8.4 MB 00:01 2023-11-08T12:08:25.536 INFO:teuthology.orchestra.run.smithi049.stdout:(10/29): python3-rados-18.0.0-7227.g25e18238.el 2.2 MB/s | 320 kB 00:00 2023-11-08T12:08:25.595 INFO:teuthology.orchestra.run.smithi049.stdout:(11/29): python3-rbd-18.0.0-7227.g25e18238.el9. 3.5 MB/s | 298 kB 00:00 2023-11-08T12:08:25.620 INFO:teuthology.orchestra.run.smithi049.stdout:(12/29): python3-rgw-18.0.0-7227.g25e18238.el9. 1.2 MB/s | 99 kB 00:00 2023-11-08T12:08:25.795 INFO:teuthology.orchestra.run.smithi049.stdout:(13/29): ceph-common-18.0.0-7227.g25e18238.el9. 7.9 MB/s | 17 MB 00:02 2023-11-08T12:08:25.921 INFO:teuthology.orchestra.run.smithi049.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 325 kB/s | 106 kB 00:00 2023-11-08T12:08:25.955 INFO:teuthology.orchestra.run.smithi049.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 476 kB/s | 159 kB 00:00 2023-11-08T12:08:25.980 INFO:teuthology.orchestra.run.smithi049.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2023-11-08T12:08:26.005 INFO:teuthology.orchestra.run.smithi049.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 661 kB 00:00 2023-11-08T12:08:26.030 INFO:teuthology.orchestra.run.smithi049.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 554 kB/s | 41 kB 00:00 2023-11-08T12:08:26.639 INFO:teuthology.orchestra.run.smithi049.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 508 kB/s | 309 kB 00:00 2023-11-08T12:08:26.739 INFO:teuthology.orchestra.run.smithi049.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 244 kB/s | 24 kB 00:00 2023-11-08T12:08:26.756 INFO:teuthology.orchestra.run.smithi049.stdout:(21/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 58 kB/s | 45 kB 00:00 2023-11-08T12:08:26.840 INFO:teuthology.orchestra.run.smithi049.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 488 kB/s | 49 kB 00:00 2023-11-08T12:08:27.190 INFO:teuthology.orchestra.run.smithi049.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 155 kB/s | 67 kB 00:00 2023-11-08T12:08:27.540 INFO:teuthology.orchestra.run.smithi049.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 547 kB/s | 192 kB 00:00 2023-11-08T12:08:27.690 INFO:teuthology.orchestra.run.smithi049.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 985 kB/s | 837 kB 00:00 2023-11-08T12:08:28.141 INFO:teuthology.orchestra.run.smithi049.stdout:(26/29): librados2-18.0.0-7227.g25e18238.el9.x8 6.9 MB/s | 3.1 MB 00:00 2023-11-08T12:08:28.349 INFO:teuthology.orchestra.run.smithi049.stdout:(27/29): librbd1-18.0.0-7227.g25e18238.el9.x86_ 15 MB/s | 3.0 MB 00:00 2023-11-08T12:08:28.924 INFO:teuthology.orchestra.run.smithi049.stdout:(28/29): thrift-0.15.0-2.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2023-11-08T12:08:29.016 INFO:teuthology.orchestra.run.smithi049.stdout:(29/29): libarrow-9.0.0-9.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:02 2023-11-08T12:08:29.021 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:08:29.021 INFO:teuthology.orchestra.run.smithi049.stdout:Total 9.4 MB/s | 54 MB 00:05 2023-11-08T12:08:29.034 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:08:29.076 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:08:29.077 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:08:29.576 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:08:29.576 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:08:30.953 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:08:31.382 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-08T12:08:31.435 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:18.0.0-7227.g25e18238.el9.x86_64 2/31 2023-11-08T12:08:31.528 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:18.0.0-7227.g25e18238.el9.x86_64 2/31 2023-11-08T12:08:31.637 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 3/31 2023-11-08T12:08:31.661 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 4/31 2023-11-08T12:08:31.724 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 4/31 2023-11-08T12:08:31.787 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/31 2023-11-08T12:08:31.816 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/31 2023-11-08T12:08:31.846 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-08T12:08:31.900 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:18.0.0-7227.g25e18238.el9. 8/31 2023-11-08T12:08:31.975 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 9/31 2023-11-08T12:08:31.998 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_6 10/31 2023-11-08T12:08:32.078 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_6 10/31 2023-11-08T12:08:32.116 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-08T12:08:32.234 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-08T12:08:32.268 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-08T12:08:32.309 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-08T12:08:32.340 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-08T12:08:32.369 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-08T12:08:32.402 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-08T12:08:32.733 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-08T12:08:32.757 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 19/31 2023-11-08T12:08:32.848 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 19/31 2023-11-08T12:08:32.878 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 20/31 2023-11-08T12:08:32.914 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-08T12:08:33.376 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-08T12:08:33.409 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 23/31 2023-11-08T12:08:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 23/31 2023-11-08T12:08:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 24/31 2023-11-08T12:08:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:18.0.0-7227.g25e18238.el9.x8 25/31 2023-11-08T12:08:35.707 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 26/31 2023-11-08T12:08:35.767 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 26/31 2023-11-08T12:08:36.558 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 26/31 2023-11-08T12:08:36.603 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 27/31 2023-11-08T12:08:36.683 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 27/31 2023-11-08T12:08:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-08T12:08:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-08T12:08:36.684 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:36.721 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 28/31 2023-11-08T12:08:46.848 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 28/31 2023-11-08T12:08:46.848 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2023-11-08T12:08:46.849 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2023-11-08T12:08:46.849 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2023-11-08T12:08:46.849 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2023-11-08T12:08:46.849 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2023-11-08T12:08:46.849 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2023-11-08T12:08:46.850 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2023-11-08T12:08:46.850 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:47.729 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 29/31 2023-11-08T12:08:47.754 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 29/31 2023-11-08T12:08:47.754 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:08:47.755 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-08T12:08:47.755 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-08T12:08:47.755 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-08T12:08:47.755 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:47.757 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-08T12:08:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-08T12:08:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-08T12:08:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-08T12:08:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 1/31 2023-11-08T12:08:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 2/31 2023-11-08T12:08:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 3/31 2023-11-08T12:08:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 4/31 2023-11-08T12:08:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 5/31 2023-11-08T12:08:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_6 6/31 2023-11-08T12:08:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 7/31 2023-11-08T12:08:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7227.g25e18238.el9. 8/31 2023-11-08T12:08:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:18.0.0-7227.g25e18238.el9.x8 9/31 2023-11-08T12:08:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 10/31 2023-11-08T12:08:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 11/31 2023-11-08T12:08:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 12/31 2023-11-08T12:08:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 13/31 2023-11-08T12:08:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-08T12:08:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-08T12:08:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/31 2023-11-08T12:08:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/31 2023-11-08T12:08:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/31 2023-11-08T12:08:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/31 2023-11-08T12:08:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-08T12:08:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-08T12:08:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-08T12:08:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-08T12:08:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-08T12:08:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-08T12:08:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-08T12:08:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-08T12:08:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:18.0.0-7227.g25e18238.el9.x86_64 28/31 2023-11-08T12:08:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-08T12:08:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 30/31 2023-11-08T12:08:50.073 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-08T12:08:50.073 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:50.073 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2023-11-08T12:08:50.074 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.074 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.074 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:08:50.074 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-08T12:08:50.074 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.075 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.075 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.075 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.075 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-08T12:08:50.075 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-08T12:08:50.076 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-08T12:08:50.076 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.076 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-08T12:08:50.076 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-08T12:08:50.076 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-08T12:08:50.077 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.077 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-08T12:08:50.077 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.077 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-08T12:08:50.077 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-08T12:08:50.078 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-08T12:08:50.078 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.078 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.078 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-08T12:08:50.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:08:50.079 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2023-11-08T12:08:50.080 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-08T12:08:50.080 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:50.080 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:08:50.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2023-11-08T12:08:51.019 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:31 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:08:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:08:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:08:51.171 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:08:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:08:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:08:51.172 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:18.0.0-7227.g25e18238.el9 ceph 117 M 2023-11-08T12:08:51.172 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:08:51.172 INFO:teuthology.orchestra.run.smithi049.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-08T12:08:51.172 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-08T12:08:51.172 INFO:teuthology.orchestra.run.smithi049.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-08T12:08:51.173 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:18.0.0-7227.g25e18238.el9 ceph 158 k 2023-11-08T12:08:51.173 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-08T12:08:51.173 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-08T12:08:51.173 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-08T12:08:51.173 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-08T12:08:51.173 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-08T12:08:51.174 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:51.174 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:08:51.174 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:08:51.174 INFO:teuthology.orchestra.run.smithi049.stdout:Install 10 Packages 2023-11-08T12:08:51.174 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:08:51.175 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 120 M 2023-11-08T12:08:51.175 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.1 G 2023-11-08T12:08:51.176 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:08:51.487 INFO:teuthology.orchestra.run.smithi049.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 186 kB 00:00 2023-11-08T12:08:51.546 INFO:teuthology.orchestra.run.smithi049.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 7.0 MB/s | 418 kB 00:00 2023-11-08T12:08:51.580 INFO:teuthology.orchestra.run.smithi049.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2023-11-08T12:08:51.613 INFO:teuthology.orchestra.run.smithi049.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2023-11-08T12:08:51.639 INFO:teuthology.orchestra.run.smithi049.stdout:(5/10): libcephsqlite-18.0.0-7227.g25e18238.el9 536 kB/s | 158 kB 00:00 2023-11-08T12:08:51.664 INFO:teuthology.orchestra.run.smithi049.stdout:(6/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2023-11-08T12:08:51.689 INFO:teuthology.orchestra.run.smithi049.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-11-08T12:08:52.056 INFO:teuthology.orchestra.run.smithi049.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 332 kB/s | 122 kB 00:00 2023-11-08T12:08:52.231 INFO:teuthology.orchestra.run.smithi049.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.2 MB/s | 1.3 MB 00:00 2023-11-08T12:08:56.789 INFO:teuthology.orchestra.run.smithi049.stdout:(10/10): ceph-test-18.0.0-7227.g25e18238.el9.x8 22 MB/s | 117 MB 00:05 2023-11-08T12:08:56.793 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:08:56.794 INFO:teuthology.orchestra.run.smithi049.stdout:Total 21 MB/s | 120 MB 00:05 2023-11-08T12:08:56.794 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:08:56.818 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:08:56.819 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:08:57.676 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:08:57.676 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:08:58.859 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:08:58.949 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-08T12:08:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-08T12:08:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-08T12:08:59.075 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-08T12:08:59.107 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-08T12:08:59.138 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-08T12:08:59.174 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-08T12:08:59.233 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-08T12:08:59.278 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 9/10 2023-11-08T12:09:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 9/10 2023-11-08T12:09:16.536 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 10/10 2023-11-08T12:09:24.643 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 10/10 2023-11-08T12:09:24.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 1/10 2023-11-08T12:09:24.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 2/10 2023-11-08T12:09:24.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-08T12:09:24.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-08T12:09:24.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-08T12:09:24.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-08T12:09:24.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-08T12:09:24.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-08T12:09:24.645 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-08T12:09:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-08T12:09:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:25.029 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:09:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:09:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-08T12:09:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-16.el9.x86_64 2023-11-08T12:09:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-08T12:09:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:09:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-08T12:09:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-08T12:09:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-08T12:09:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-08T12:09:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-08T12:09:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:25.032 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:09:25.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2023-11-08T12:09:25.918 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:05 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:09:26.076 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:09:26.078 INFO:teuthology.orchestra.run.smithi049.stdout:========================================================================================= 2023-11-08T12:09:26.078 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:09:26.078 INFO:teuthology.orchestra.run.smithi049.stdout:========================================================================================= 2023-11-08T12:09:26.078 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:09:26.079 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:18.0.0-7227.g25e18238.el9 ceph 6.5 k 2023-11-08T12:09:26.079 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:09:26.079 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:18.0.0-7227.g25e18238.el9 ceph 2.0 M 2023-11-08T12:09:26.079 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:18.0.0-7227.g25e18238.el9 ceph 1.5 M 2023-11-08T12:09:26.080 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 243 k 2023-11-08T12:09:26.080 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:18.0.0-7227.g25e18238.el9 ceph 8.9 M 2023-11-08T12:09:26.080 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:18.0.0-7227.g25e18238.el9 ceph 117 M 2023-11-08T12:09:26.080 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-08T12:09:26.080 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-08T12:09:26.081 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-08T12:09:26.081 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-08T12:09:26.081 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-08T12:09:26.081 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-08T12:09:26.082 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-08T12:09:26.082 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-08T12:09:26.082 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-08T12:09:26.082 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-08T12:09:26.082 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-08T12:09:26.083 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-08T12:09:26.083 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-08T12:09:26.083 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-08T12:09:26.083 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-08T12:09:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-08T12:09:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-08T12:09:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-08T12:09:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-08T12:09:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-08T12:09:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-08T12:09:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-08T12:09:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-08T12:09:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-08T12:09:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-08T12:09:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-08T12:09:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-08T12:09:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:26.087 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:09:26.088 INFO:teuthology.orchestra.run.smithi049.stdout:========================================================================================= 2023-11-08T12:09:26.088 INFO:teuthology.orchestra.run.smithi049.stdout:Install 33 Packages 2023-11-08T12:09:26.088 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:26.089 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 133 M 2023-11-08T12:09:26.090 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.4 G 2023-11-08T12:09:26.090 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:09:26.751 INFO:teuthology.orchestra.run.smithi049.stdout:(1/33): ceph-18.0.0-7227.g25e18238.el9.x86_64.r 30 kB/s | 6.5 kB 00:00 2023-11-08T12:09:27.068 INFO:teuthology.orchestra.run.smithi049.stdout:(2/33): ceph-mds-18.0.0-7227.g25e18238.el9.x86_ 3.7 MB/s | 2.0 MB 00:00 2023-11-08T12:09:27.260 INFO:teuthology.orchestra.run.smithi049.stdout:(3/33): ceph-mgr-18.0.0-7227.g25e18238.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2023-11-08T12:09:27.368 INFO:teuthology.orchestra.run.smithi049.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7227.g25e1 2.2 MB/s | 243 kB 00:00 2023-11-08T12:09:27.468 INFO:teuthology.orchestra.run.smithi049.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-08T12:09:27.502 INFO:teuthology.orchestra.run.smithi049.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2023-11-08T12:09:27.535 INFO:teuthology.orchestra.run.smithi049.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-08T12:09:27.577 INFO:teuthology.orchestra.run.smithi049.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-11-08T12:09:27.611 INFO:teuthology.orchestra.run.smithi049.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-08T12:09:27.645 INFO:teuthology.orchestra.run.smithi049.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-11-08T12:09:27.678 INFO:teuthology.orchestra.run.smithi049.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-08T12:09:27.721 INFO:teuthology.orchestra.run.smithi049.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-11-08T12:09:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:(13/33): ceph-mon-18.0.0-7227.g25e18238.el9.x86 8.3 MB/s | 8.9 MB 00:01 2023-11-08T12:09:28.038 INFO:teuthology.orchestra.run.smithi049.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 236 kB/s | 51 kB 00:00 2023-11-08T12:09:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2023-11-08T12:09:28.655 INFO:teuthology.orchestra.run.smithi049.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 644 kB/s | 172 kB 00:00 2023-11-08T12:09:28.805 INFO:teuthology.orchestra.run.smithi049.stdout:(17/33): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2023-11-08T12:09:29.006 INFO:teuthology.orchestra.run.smithi049.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2023-11-08T12:09:29.106 INFO:teuthology.orchestra.run.smithi049.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 796 kB/s | 358 kB 00:00 2023-11-08T12:09:29.131 INFO:teuthology.orchestra.run.smithi049.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2023-11-08T12:09:29.206 INFO:teuthology.orchestra.run.smithi049.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2023-11-08T12:09:29.240 INFO:teuthology.orchestra.run.smithi049.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2023-11-08T12:09:29.340 INFO:teuthology.orchestra.run.smithi049.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2023-11-08T12:09:29.449 INFO:teuthology.orchestra.run.smithi049.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 222 kB/s | 46 kB 00:00 2023-11-08T12:09:29.483 INFO:teuthology.orchestra.run.smithi049.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 555 kB/s | 79 kB 00:00 2023-11-08T12:09:29.625 INFO:teuthology.orchestra.run.smithi049.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 116 kB/s | 16 kB 00:00 2023-11-08T12:09:29.808 INFO:teuthology.orchestra.run.smithi049.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 759 kB/s | 272 kB 00:00 2023-11-08T12:09:29.834 INFO:teuthology.orchestra.run.smithi049.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 431 kB/s | 90 kB 00:00 2023-11-08T12:09:29.909 INFO:teuthology.orchestra.run.smithi049.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2023-11-08T12:09:30.117 INFO:teuthology.orchestra.run.smithi049.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 810 kB/s | 230 kB 00:00 2023-11-08T12:09:30.217 INFO:teuthology.orchestra.run.smithi049.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2023-11-08T12:09:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.0 MB/s | 427 kB 00:00 2023-11-08T12:09:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:(33/33): ceph-osd-18.0.0-7227.g25e18238.el9.x86 22 MB/s | 117 MB 00:05 2023-11-08T12:09:32.472 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:09:32.472 INFO:teuthology.orchestra.run.smithi049.stdout:Total 21 MB/s | 133 MB 00:06 2023-11-08T12:09:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:09:32.552 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:09:32.552 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:09:35.453 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:09:35.521 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-08T12:09:35.561 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-08T12:09:35.594 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-08T12:09:35.690 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-08T12:09:35.783 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-08T12:09:35.831 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-08T12:09:35.868 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-08T12:09:35.939 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-08T12:09:35.991 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-08T12:09:36.028 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-08T12:09:36.079 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-08T12:09:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-08T12:09:36.363 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-08T12:09:36.416 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-08T12:09:36.460 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-08T12:09:36.497 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-08T12:09:36.536 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-08T12:09:36.622 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-08T12:09:36.671 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-08T12:09:36.705 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-08T12:09:36.746 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-08T12:09:36.781 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-08T12:09:36.874 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-08T12:09:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-08T12:09:36.986 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-08T12:09:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-08T12:09:37.251 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-08T12:09:37.418 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7227.g25e18238.el9. 28/33 2023-11-08T12:09:37.458 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 29/33 2023-11-08T12:09:37.485 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 29/33 2023-11-08T12:09:37.485 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:09:37.485 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-08T12:09:37.486 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-08T12:09:37.486 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-08T12:09:37.486 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:56.824 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 30/33 2023-11-08T12:09:56.851 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 30/33 2023-11-08T12:09:56.851 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:09:56.852 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-08T12:09:56.852 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-08T12:09:56.852 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-08T12:09:56.852 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:57.782 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 31/33 2023-11-08T12:09:57.808 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 31/33 2023-11-08T12:09:57.808 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:09:57.808 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-08T12:09:57.809 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-08T12:09:57.809 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-08T12:09:57.809 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:58.025 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 32/33 2023-11-08T12:09:58.049 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 32/33 2023-11-08T12:09:58.049 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:09:58.050 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-08T12:09:58.050 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-08T12:09:58.050 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-08T12:09:58.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:09:58.116 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:18.0.0-7227.g25e18238.el9.x86_64 33/33 2023-11-08T12:10:08.168 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:18.0.0-7227.g25e18238.el9.x86_64 33/33 2023-11-08T12:10:08.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:18.0.0-7227.g25e18238.el9.x86_64 1/33 2023-11-08T12:10:08.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 2/33 2023-11-08T12:10:08.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 3/33 2023-11-08T12:10:08.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 4/33 2023-11-08T12:10:08.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 5/33 2023-11-08T12:10:08.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7227.g25e18238.el9. 6/33 2023-11-08T12:10:08.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-08T12:10:08.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-08T12:10:08.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-08T12:10:08.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-08T12:10:08.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-08T12:10:08.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-08T12:10:08.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-08T12:10:08.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-08T12:10:08.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-08T12:10:08.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-08T12:10:08.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-08T12:10:08.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-08T12:10:08.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-08T12:10:08.173 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-08T12:10:08.173 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-08T12:10:08.173 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-08T12:10:08.173 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-08T12:10:08.173 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-08T12:10:08.174 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-08T12:10:08.174 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-08T12:10:08.174 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-08T12:10:08.174 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-08T12:10:08.174 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-08T12:10:08.175 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-08T12:10:08.175 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-08T12:10:08.176 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-08T12:10:09.068 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-08T12:10:09.068 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:09.069 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:09.069 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:09.069 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:09.069 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:09.069 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:09.070 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:09.070 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:09.070 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-08T12:10:09.070 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-08T12:10:09.070 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-08T12:10:09.071 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-08T12:10:09.071 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-08T12:10:09.071 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-08T12:10:09.071 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-08T12:10:09.071 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-08T12:10:09.072 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-08T12:10:09.072 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-08T12:10:09.072 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-08T12:10:09.072 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-08T12:10:09.072 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-08T12:10:09.073 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-08T12:10:09.073 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-08T12:10:09.073 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2023-11-08T12:10:09.073 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-08T12:10:09.073 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-08T12:10:09.074 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-08T12:10:09.074 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-08T12:10:09.074 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-08T12:10:09.074 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-08T12:10:09.075 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-08T12:10:09.075 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-08T12:10:09.075 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-08T12:10:09.075 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-08T12:10:09.075 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-08T12:10:09.076 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:09.076 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:09.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2023-11-08T12:10:09.908 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:49 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:10.005 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:10.057 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:10.058 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:10.059 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:10.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2023-11-08T12:10:10.724 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:50 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:10.872 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:10.873 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:10.873 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:10.874 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:10.874 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:10.874 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 768 k 2023-11-08T12:10:10.874 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:10.874 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:10.875 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:10.875 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:10:10.875 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:10.875 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 768 k 2023-11-08T12:10:10.875 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 782 k 2023-11-08T12:10:10.876 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:11.298 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-18.0.0-7227.g25e18238.el9.noarch.rpm 1.8 MB/s | 768 kB 00:00 2023-11-08T12:10:11.299 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:11.299 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.8 MB/s | 768 kB 00:00 2023-11-08T12:10:11.300 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:11.304 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:11.305 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:11.310 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:11.311 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:11.384 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:11.836 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:11.930 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:12.317 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:12.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:12.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:12.628 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:12.628 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:12.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:12.629 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:12.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2023-11-08T12:10:13.455 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:53 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:13.603 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:13.604 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:13.604 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-11-08T12:10:13.605 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:13.605 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:13.605 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7227.g25e18238.el9 ceph 143 k 2023-11-08T12:10:13.605 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:13.605 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:13.606 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:13.606 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:10:13.606 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:13.606 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 143 k 2023-11-08T12:10:13.607 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 438 k 2023-11-08T12:10:13.607 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:13.829 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-18.0.0-7227.g25e182 642 kB/s | 143 kB 00:00 2023-11-08T12:10:13.830 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:13.830 INFO:teuthology.orchestra.run.smithi049.stdout:Total 638 kB/s | 143 kB 00:00 2023-11-08T12:10:13.830 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:13.838 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:13.838 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:13.892 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:13.893 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:14.113 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7227.g25e18238. 1/1 2023-11-08T12:10:14.139 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7227.g25e18238. 1/1 2023-11-08T12:10:14.139 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:10:14.139 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-08T12:10:14.140 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:15.145 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7227.g25e18238. 1/1 2023-11-08T12:10:15.145 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:15.145 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:15.145 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:15.146 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:15.146 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:15.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2023-11-08T12:10:15.975 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:55 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:16.073 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:16.125 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:16.126 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:16.126 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:16.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2023-11-08T12:10:16.787 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:56 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:16.939 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:16.939 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:16.940 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:16.940 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 1.8 M 2023-11-08T12:10:16.940 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:10:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 23 k 2023-11-08T12:10:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 15 k 2023-11-08T12:10:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-08T12:10:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-08T12:10:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-08T12:10:16.941 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2023-11-08T12:10:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-08T12:10:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:16.942 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:16.942 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:16.942 INFO:teuthology.orchestra.run.smithi049.stdout:Install 7 Packages 2023-11-08T12:10:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:16.943 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.1 M 2023-11-08T12:10:16.943 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 8.6 M 2023-11-08T12:10:16.944 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:17.307 INFO:teuthology.orchestra.run.smithi049.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7227.g25e1 77 kB/s | 15 kB 00:00 2023-11-08T12:10:17.349 INFO:teuthology.orchestra.run.smithi049.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7227.g25e 97 kB/s | 23 kB 00:00 2023-11-08T12:10:17.495 INFO:teuthology.orchestra.run.smithi049.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 49 kB/s | 9.0 kB 00:00 2023-11-08T12:10:17.533 INFO:teuthology.orchestra.run.smithi049.stdout:(4/7): ceph-mgr-dashboard-18.0.0-7227.g25e18238 4.2 MB/s | 1.8 MB 00:00 2023-11-08T12:10:17.667 INFO:teuthology.orchestra.run.smithi049.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 175 kB/s | 31 kB 00:00 2023-11-08T12:10:17.684 INFO:teuthology.orchestra.run.smithi049.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 122 kB/s | 41 kB 00:00 2023-11-08T12:10:18.009 INFO:teuthology.orchestra.run.smithi049.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 396 kB/s | 188 kB 00:00 2023-11-08T12:10:18.015 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:18.015 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.9 MB/s | 2.1 MB 00:01 2023-11-08T12:10:18.015 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:18.021 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:18.022 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:18.100 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:18.101 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:18.409 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:18.477 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-08T12:10:18.511 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-08T12:10:18.558 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-08T12:10:18.589 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-08T12:10:18.618 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7227.g25e18238.el9.n 5/7 2023-11-08T12:10:18.865 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7227.g25e18238.el9. 6/7 2023-11-08T12:10:18.896 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarc 7/7 2023-11-08T12:10:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarc 7/7 2023-11-08T12:10:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7227.g25e18238.el9. 1/7 2023-11-08T12:10:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarc 2/7 2023-11-08T12:10:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7227.g25e18238.el9.n 3/7 2023-11-08T12:10:19.463 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-08T12:10:19.463 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-08T12:10:19.464 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-08T12:10:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-08T12:10:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:19.798 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:19.799 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:19.799 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:19.800 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-08T12:10:19.800 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-08T12:10:19.800 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-08T12:10:19.801 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-08T12:10:19.801 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:19.801 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:19.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-08T12:10:20.586 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:00 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:20.739 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:20.741 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2023-11-08T12:10:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:20.741 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2023-11-08T12:10:20.741 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 7.4 M 2023-11-08T12:10:20.742 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:10:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-08T12:10:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-08T12:10:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-08T12:10:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-08T12:10:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-08T12:10:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-08T12:10:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-08T12:10:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-08T12:10:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-08T12:10:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-08T12:10:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-08T12:10:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:20.744 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:20.745 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2023-11-08T12:10:20.745 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2023-11-08T12:10:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:20.746 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 37 M 2023-11-08T12:10:20.746 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 209 M 2023-11-08T12:10:20.746 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:20.873 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 243 kB/s | 32 kB 00:00 2023-11-08T12:10:20.907 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2023-11-08T12:10:20.940 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-11-08T12:10:21.007 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-11-08T12:10:21.066 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.5 MB/s | 205 kB 00:00 2023-11-08T12:10:21.141 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-11-08T12:10:21.217 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-11-08T12:10:21.242 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-11-08T12:10:21.334 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 8.8 MB/s | 803 kB 00:00 2023-11-08T12:10:21.493 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2023-11-08T12:10:21.518 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-11-08T12:10:22.047 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 5.7 MB/s | 7.4 MB 00:01 2023-11-08T12:10:22.047 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:22.047 INFO:teuthology.orchestra.run.smithi049.stdout:Total 28 MB/s | 37 MB 00:01 2023-11-08T12:10:22.048 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:22.070 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:22.070 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:22.507 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:22.508 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:23.234 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:23.308 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-08T12:10:23.339 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-08T12:10:23.449 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-08T12:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-08T12:10:23.685 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-08T12:10:23.733 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-08T12:10:23.824 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-08T12:10:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-08T12:10:24.633 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-08T12:10:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-08T12:10:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-08T12:10:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18 12/12 2023-11-08T12:10:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18 12/12 2023-11-08T12:10:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18 1/12 2023-11-08T12:10:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-08T12:10:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-08T12:10:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-08T12:10:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-08T12:10:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-08T12:10:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-08T12:10:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-08T12:10:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-08T12:10:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-08T12:10:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-08T12:10:30.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-08T12:10:30.299 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:30.299 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:30.299 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:30.300 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-08T12:10:30.300 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-08T12:10:30.300 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-08T12:10:30.300 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-08T12:10:30.300 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-08T12:10:30.301 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-08T12:10:30.301 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-08T12:10:30.301 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-08T12:10:30.301 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-08T12:10:30.301 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-08T12:10:30.302 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-08T12:10:30.302 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:30.302 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:30.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2023-11-08T12:10:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:11 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:31.440 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:31.441 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-11-08T12:10:31.441 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:31.442 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-11-08T12:10:31.442 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:31.442 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 50 k 2023-11-08T12:10:31.442 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:10:31.442 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-08T12:10:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-08T12:10:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-08T12:10:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-08T12:10:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-08T12:10:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-08T12:10:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-08T12:10:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-08T12:10:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-08T12:10:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-08T12:10:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-08T12:10:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-08T12:10:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:31.445 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:31.445 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-11-08T12:10:31.445 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2023-11-08T12:10:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:31.446 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.0 M 2023-11-08T12:10:31.446 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2023-11-08T12:10:31.447 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:31.733 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 214 kB/s | 24 kB 00:00 2023-11-08T12:10:31.759 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 129 kB/s | 18 kB 00:00 2023-11-08T12:10:31.800 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-11-08T12:10:31.826 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-11-08T12:10:31.851 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-08T12:10:31.876 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2023-11-08T12:10:32.094 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): ceph-mgr-rook-18.0.0-7227.g25e18238.el9 106 kB/s | 50 kB 00:00 2023-11-08T12:10:32.227 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2023-11-08T12:10:32.269 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 77 kB/s | 32 kB 00:00 2023-11-08T12:10:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 296 kB/s | 59 kB 00:00 2023-11-08T12:10:32.661 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 468 kB/s | 90 kB 00:00 2023-11-08T12:10:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-google-auth-2.23.3-1.el9.noarc 333 kB/s | 197 kB 00:00 2023-11-08T12:10:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-11-08T12:10:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-11-08T12:10:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:32.775 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:32.775 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:32.889 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:32.890 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:33.422 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:33.625 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-08T12:10:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-08T12:10:33.745 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-08T12:10:33.798 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-08T12:10:33.839 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-08T12:10:33.936 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-08T12:10:34.056 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-08T12:10:34.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-08T12:10:34.872 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-08T12:10:34.929 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-08T12:10:34.963 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-08T12:10:35.020 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-08T12:10:35.049 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 13/13 2023-11-08T12:10:35.822 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 13/13 2023-11-08T12:10:35.822 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 1/13 2023-11-08T12:10:35.822 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-08T12:10:35.823 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-08T12:10:35.823 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-08T12:10:35.823 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-08T12:10:35.823 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-08T12:10:35.823 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-08T12:10:35.824 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-08T12:10:35.824 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-08T12:10:35.824 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-08T12:10:35.824 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-08T12:10:35.824 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-08T12:10:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-08T12:10:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-08T12:10:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-08T12:10:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-08T12:10:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-08T12:10:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-08T12:10:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-08T12:10:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-08T12:10:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-08T12:10:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-08T12:10:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-08T12:10:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-08T12:10:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-08T12:10:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:36.207 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:36.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2023-11-08T12:10:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:17 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:37.204 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:37.205 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-11-08T12:10:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:37.205 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-11-08T12:10:37.205 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 132 k 2023-11-08T12:10:37.206 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:10:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-08T12:10:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-08T12:10:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-08T12:10:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-08T12:10:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-08T12:10:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:37.207 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:37.207 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-11-08T12:10:37.207 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-11-08T12:10:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:37.208 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 6.8 M 2023-11-08T12:10:37.209 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 33 M 2023-11-08T12:10:37.209 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:37.422 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-11-08T12:10:37.538 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-11-08T12:10:37.580 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7227.g25e18238.e 413 kB/s | 132 kB 00:00 2023-11-08T12:10:37.822 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 204 kB/s | 58 kB 00:00 2023-11-08T12:10:37.865 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2023-11-08T12:10:38.098 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 811 kB/s | 548 kB 00:00 2023-11-08T12:10:38.100 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:38.100 INFO:teuthology.orchestra.run.smithi049.stdout:Total 7.6 MB/s | 6.8 MB 00:00 2023-11-08T12:10:38.100 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:38.112 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:38.112 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:38.212 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:38.213 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:38.520 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:38.697 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-08T12:10:38.750 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-08T12:10:39.317 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-08T12:10:39.397 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-08T12:10:39.464 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-08T12:10:39.496 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 6/6 2023-11-08T12:10:40.309 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 6/6 2023-11-08T12:10:40.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/6 2023-11-08T12:10:40.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-08T12:10:40.310 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-08T12:10:40.310 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-08T12:10:40.310 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-08T12:10:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-08T12:10:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:40.643 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-08T12:10:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-08T12:10:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-08T12:10:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-08T12:10:40.645 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-08T12:10:40.645 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:40.645 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:40.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2023-11-08T12:10:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:21 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:41.625 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:41.626 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:41.627 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:41.627 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:41.627 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:41.627 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:18.0.0-7227.g25e18238.el9 ceph 833 k 2023-11-08T12:10:41.627 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:10:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-08T12:10:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:41.628 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:41.628 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:41.628 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2023-11-08T12:10:41.629 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:41.629 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 911 k 2023-11-08T12:10:41.629 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 3.0 M 2023-11-08T12:10:41.629 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:41.750 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 644 kB/s | 79 kB 00:00 2023-11-08T12:10:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-fuse-18.0.0-7227.g25e18238.el9.x86_ 1.0 MB/s | 833 kB 00:00 2023-11-08T12:10:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:42.460 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.1 MB/s | 911 kB 00:00 2023-11-08T12:10:42.460 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:42.485 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:42.486 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:42.569 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:42.570 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:42.765 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:42.920 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-08T12:10:43.002 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 2/2 2023-11-08T12:10:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 2/2 2023-11-08T12:10:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/2 2023-11-08T12:10:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-08T12:10:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:44.002 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-08T12:10:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:44.003 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:44.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2023-11-08T12:10:44.872 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:24 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:45.021 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:45.022 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:10:45.023 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:45.023 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 262 k 2023-11-08T12:10:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:45.024 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:45.024 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:45.024 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:10:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:45.024 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 262 k 2023-11-08T12:10:45.025 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.2 M 2023-11-08T12:10:45.025 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:45.698 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-volume-18.0.0-7227.g25e18238.el9.noarch.rp 388 kB/s | 262 kB 00:00 2023-11-08T12:10:45.699 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:45.699 INFO:teuthology.orchestra.run.smithi049.stdout:Total 387 kB/s | 262 kB 00:00 2023-11-08T12:10:45.700 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:45.708 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:45.708 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:45.755 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:45.755 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:45.979 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:46.067 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:46.083 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:46.084 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:10:46.084 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-08T12:10:46.084 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:47.123 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:10:47.123 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:47.123 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:47.123 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:10:47.124 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:47.124 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:47.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2023-11-08T12:10:48.022 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:28 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:48.174 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:48.175 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:48.175 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-11-08T12:10:48.175 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:48.176 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:48.176 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 127 k 2023-11-08T12:10:48.176 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:48.176 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:48.176 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:48.177 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:10:48.177 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:48.177 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 127 k 2023-11-08T12:10:48.177 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 460 k 2023-11-08T12:10:48.177 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:48.370 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-18.0.0-7227.g25e18238.el9.x86_64 653 kB/s | 127 kB 00:00 2023-11-08T12:10:48.371 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:48.371 INFO:teuthology.orchestra.run.smithi049.stdout:Total 647 kB/s | 127 kB 00:00 2023-11-08T12:10:48.372 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:48.377 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:48.377 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:48.436 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:48.437 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:48.586 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:48.696 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:49.175 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:49.491 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:49.491 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:49.491 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:49.492 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:49.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:49.492 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:49.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2023-11-08T12:10:50.338 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:30 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:50.437 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:50.490 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:50.491 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:50.492 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:50.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2023-11-08T12:10:51.161 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:31 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:51.312 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:51.313 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:51.313 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-11-08T12:10:51.314 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:51.314 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:51.314 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 31 k 2023-11-08T12:10:51.314 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:51.314 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:51.315 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:51.315 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:10:51.315 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:51.315 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 31 k 2023-11-08T12:10:51.315 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 139 k 2023-11-08T12:10:51.316 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:51.557 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-18.0.0-7227.g25e18238.el9.x86_6 128 kB/s | 31 kB 00:00 2023-11-08T12:10:51.558 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:51.558 INFO:teuthology.orchestra.run.smithi049.stdout:Total 127 kB/s | 31 kB 00:00 2023-11-08T12:10:51.559 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:51.562 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:51.562 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:51.592 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:51.593 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:51.681 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:51.808 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:52.269 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:52.554 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:52.555 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:52.555 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:52.555 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:52.555 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:52.555 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:52.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2023-11-08T12:10:53.330 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:33 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:53.428 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:53.481 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:53.482 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:53.482 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:53.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2023-11-08T12:10:54.148 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:34 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:54.246 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:54.302 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:54.302 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:54.303 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:54.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2023-11-08T12:10:54.974 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:34 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:55.072 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:55.126 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:55.127 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:55.127 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:55.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2023-11-08T12:10:55.807 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:35 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:55.903 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:55.957 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:55.958 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:55.959 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:56.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2023-11-08T12:10:56.636 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:36 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:56.734 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:56.788 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:56.789 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:56.789 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:56.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2023-11-08T12:10:57.471 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:37 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:57.572 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:10:57.629 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:57.629 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-08T12:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:57.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2023-11-08T12:10:58.301 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:38 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:10:58.456 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:10:58.457 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:58.457 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-08T12:10:58.457 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:58.458 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:10:58.458 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:18.0.0-7227.g25e18238.el9 ceph 86 k 2023-11-08T12:10:58.458 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:58.458 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:10:58.458 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:10:58.459 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:10:58.459 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:58.459 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 86 k 2023-11-08T12:10:58.459 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 256 k 2023-11-08T12:10:58.459 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:10:58.653 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-18.0.0-7227.g25e18238.el9.x86_64.rpm 441 kB/s | 86 kB 00:00 2023-11-08T12:10:58.653 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:10:58.654 INFO:teuthology.orchestra.run.smithi049.stdout:Total 438 kB/s | 86 kB 00:00 2023-11-08T12:10:58.654 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:10:58.661 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:10:58.662 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:10:58.720 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:10:58.721 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:10:58.871 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:10:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:59.409 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:59.712 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:10:59.713 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:10:59.713 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:10:59.713 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:10:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:10:59.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2023-11-08T12:11:00.553 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:40 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:11:00.704 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:11:00.705 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:00.705 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-11-08T12:11:00.705 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:00.705 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:11:00.706 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:18.0.0-7227.g25e18238.el9 ceph 3.0 M 2023-11-08T12:11:00.706 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:00.706 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:11:00.706 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:00.706 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:11:00.707 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:00.707 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.0 M 2023-11-08T12:11:00.707 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 13 M 2023-11-08T12:11:00.707 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:11:01.247 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-18.0.0-7227.g25e18238.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2023-11-08T12:11:01.248 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:11:01.248 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.5 MB/s | 3.0 MB 00:00 2023-11-08T12:11:01.248 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:11:01.258 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:11:01.258 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:11:01.336 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:11:01.336 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:11:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:11:01.847 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:11:01.873 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:11:01.873 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:11:01.874 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-08T12:11:01.874 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-08T12:11:01.874 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-08T12:11:01.874 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:02.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:11:02.995 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:02.995 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:11:02.995 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:11:02.995 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:02.996 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:11:03.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2023-11-08T12:11:03.857 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:43 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:11:04.008 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:11:04.009 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:04.010 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-08T12:11:04.010 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:04.010 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:11:04.010 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:18.0.0-7227.g25e18238.el9 ceph 169 k 2023-11-08T12:11:04.010 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:04.011 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:11:04.011 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:04.011 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-08T12:11:04.011 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:04.011 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 169 k 2023-11-08T12:11:04.012 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 537 k 2023-11-08T12:11:04.012 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:11:04.609 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-18.0.0-7227.g25e18238.el9.x86_64.rpm 282 kB/s | 169 kB 00:00 2023-11-08T12:11:04.610 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:11:04.610 INFO:teuthology.orchestra.run.smithi049.stdout:Total 282 kB/s | 169 kB 00:00 2023-11-08T12:11:04.610 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:11:04.619 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:11:04.619 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:11:04.679 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:11:04.679 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:11:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:11:05.002 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:11:05.480 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:11:05.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:11:05.770 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:05.771 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:11:05.771 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:11:05.771 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:05.771 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:11:05.989 DEBUG:teuthology.parallel:result is None 2023-11-08T12:11:05.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=25e18238395a1cded51f624e1e84ec38cbcb1b4a 2023-11-08T12:11:06.139 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-08T12:11:06.162 INFO:teuthology.orchestra.run.smithi049.stdout:18.0.0-7227.g25e18238.el9 2023-11-08T12:11:06.163 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7227.g25e18238.el9 2023-11-08T12:11:06.164 INFO:teuthology.task.install:The correct ceph version 18.0.0-7227.g25e18238 is installed. 2023-11-08T12:11:06.164 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-08T12:11:06.165 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:06.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-08T12:11:06.234 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-08T12:11:06.235 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:06.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2023-11-08T12:11:06.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-08T12:11:06.376 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-08T12:11:06.376 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:06.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-08T12:11:06.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-08T12:11:06.516 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-08T12:11:06.516 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:06.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2023-11-08T12:11:06.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-08T12:11:06.657 INFO:teuthology.run_tasks:Running task ceph... 2023-11-08T12:11:06.735 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-08T12:11:06.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2023-11-08T12:11:06.767 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-08T12:11:06.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-08T12:11:06.838 INFO:tasks.ceph:Creating extra log directories... 2023-11-08T12:11:06.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-08T12:11:06.917 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-08T12:11:06.917 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-08T12:11:06.918 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916', 'branch': 'wip-matanb-crimson-do_osd_ops_execute-v3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read}', 'email': None, 'first_in_suite': False, 'job_id': '7451916', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-do_osd_ops_execute-v3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'db6fbc96537be451d7eb9f6302778fac84588cb6'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6890, 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'db6fbc96537be451d7eb9f6302778fac84588cb6', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMPwXdIR+bESh0S4xXdY2pTg+iFop4Ey4+AAMLzCrhE5j9QdSGdCam+TJ2R+WU2BKgSobadmnlK8/EEWU3mI4zg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-08_11:46:17', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714'} 2023-11-08T12:11:06.919 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-08T12:11:06.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T12:11:07.037 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:07.037 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2023-11-08T12:11:07.094 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-08T12:11:07.095 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2023-11-08T12:11:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-08T12:11:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:11:07.150 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-11-08T12:11:07.150 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:11:07.151 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:11:07.151 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-11-08 12:11:02.112119055 +0000 2023-11-08T12:11:07.151 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-11-08 12:07:32.855783279 +0000 2023-11-08T12:11:07.151 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-11-08 12:07:32.855783279 +0000 2023-11-08T12:11:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-11-08 12:07:32.855783279 +0000 2023-11-08T12:11:07.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-08T12:11:07.218 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-11-08T12:11:07.219 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-11-08T12:11:07.219 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000106167 s, 4.8 MB/s 2023-11-08T12:11:07.220 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-08T12:11:07.277 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2023-11-08T12:11:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-08T12:11:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:11:07.334 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-11-08T12:11:07.334 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:11:07.334 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:11:07.334 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-11-08 12:11:02.113119027 +0000 2023-11-08T12:11:07.334 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-11-08 12:07:33.360768838 +0000 2023-11-08T12:11:07.335 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-11-08 12:07:33.360768838 +0000 2023-11-08T12:11:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-11-08 12:07:33.360768838 +0000 2023-11-08T12:11:07.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-08T12:11:07.401 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-11-08T12:11:07.402 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-11-08T12:11:07.402 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 8.3857e-05 s, 6.1 MB/s 2023-11-08T12:11:07.403 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-08T12:11:07.460 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2023-11-08T12:11:07.515 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-08T12:11:07.515 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:11:07.515 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-11-08T12:11:07.516 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:11:07.516 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:11:07.517 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-11-08 12:11:02.113119027 +0000 2023-11-08T12:11:07.517 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-11-08 12:07:33.853754740 +0000 2023-11-08T12:11:07.517 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-11-08 12:07:33.853754740 +0000 2023-11-08T12:11:07.517 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-11-08 12:07:33.853754740 +0000 2023-11-08T12:11:07.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-08T12:11:07.584 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-11-08T12:11:07.584 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-11-08T12:11:07.585 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.4777e-05 s, 5.4 MB/s 2023-11-08T12:11:07.586 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-08T12:11:07.641 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2023-11-08T12:11:07.697 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-08T12:11:07.697 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:11:07.697 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-11-08T12:11:07.697 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:11:07.698 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:11:07.698 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-11-08 12:11:02.113119027 +0000 2023-11-08T12:11:07.698 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-11-08 12:07:34.339740842 +0000 2023-11-08T12:11:07.698 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-11-08 12:07:34.339740842 +0000 2023-11-08T12:11:07.698 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-11-08 12:07:34.339740842 +0000 2023-11-08T12:11:07.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-08T12:11:07.766 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-11-08T12:11:07.766 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-11-08T12:11:07.767 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.7755e-05 s, 5.2 MB/s 2023-11-08T12:11:07.768 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-08T12:11:07.824 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:11:07.824 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T12:11:07.824 INFO:tasks.ceph:Generating config... 2023-11-08T12:11:07.825 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-08T12:11:07.826 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-08T12:11:07.826 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-08T12:11:07.826 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-08T12:11:07.827 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-08T12:11:07.827 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-08T12:11:07.827 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-08T12:11:07.827 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-08T12:11:07.828 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-08T12:11:07.828 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-11-08T12:11:07.828 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-11-08T12:11:07.828 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-11-08T12:11:07.829 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-11-08T12:11:07.829 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-08T12:11:07.829 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-08T12:11:07.830 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-08T12:11:07.830 INFO:tasks.ceph:[osd] log to file = False 2023-11-08T12:11:07.830 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-08T12:11:07.830 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-08T12:11:07.831 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-08T12:11:07.831 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-08T12:11:07.831 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-08T12:11:07.831 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-08T12:11:07.832 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-08T12:11:07.832 INFO:tasks.ceph:[global] auth client required = none 2023-11-08T12:11:07.832 INFO:tasks.ceph:[global] auth cluster required = none 2023-11-08T12:11:07.832 INFO:tasks.ceph:[global] auth service required = none 2023-11-08T12:11:07.833 INFO:tasks.ceph:[global] auth supported = none 2023-11-08T12:11:07.833 INFO:tasks.ceph:[global] debug asok = 0/0 2023-11-08T12:11:07.833 INFO:tasks.ceph:[global] debug auth = 0/0 2023-11-08T12:11:07.834 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-11-08T12:11:07.834 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-11-08T12:11:07.834 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-11-08T12:11:07.835 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-11-08T12:11:07.835 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-11-08T12:11:07.835 INFO:tasks.ceph:[global] debug client = 0/0 2023-11-08T12:11:07.835 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-11-08T12:11:07.836 INFO:tasks.ceph:[global] debug context = 0/0 2023-11-08T12:11:07.836 INFO:tasks.ceph:[global] debug crush = 0/0 2023-11-08T12:11:07.836 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-11-08T12:11:07.836 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-11-08T12:11:07.837 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-11-08T12:11:07.837 INFO:tasks.ceph:[global] debug filer = 0/0 2023-11-08T12:11:07.837 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-11-08T12:11:07.837 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-11-08T12:11:07.838 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-11-08T12:11:07.838 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-11-08T12:11:07.838 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-11-08T12:11:07.839 INFO:tasks.ceph:[global] debug journal = 0/0 2023-11-08T12:11:07.839 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-11-08T12:11:07.839 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-11-08T12:11:07.839 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-11-08T12:11:07.840 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-11-08T12:11:07.840 INFO:tasks.ceph:[global] debug mds = 0/0 2023-11-08T12:11:07.840 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-11-08T12:11:07.840 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-11-08T12:11:07.841 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-11-08T12:11:07.841 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-11-08T12:11:07.841 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-11-08T12:11:07.841 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-11-08T12:11:07.842 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-11-08T12:11:07.842 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-11-08T12:11:07.842 INFO:tasks.ceph:[global] debug mon = 0/0 2023-11-08T12:11:07.842 INFO:tasks.ceph:[global] debug monc = 0/0 2023-11-08T12:11:07.843 INFO:tasks.ceph:[global] debug ms = 0/0 2023-11-08T12:11:07.843 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-11-08T12:11:07.843 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-11-08T12:11:07.843 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-11-08T12:11:07.844 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-11-08T12:11:07.844 INFO:tasks.ceph:[global] debug osd = 0/0 2023-11-08T12:11:07.844 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-11-08T12:11:07.844 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-11-08T12:11:07.845 INFO:tasks.ceph:[global] debug rados = 0/0 2023-11-08T12:11:07.845 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-11-08T12:11:07.845 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-11-08T12:11:07.845 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-11-08T12:11:07.846 INFO:tasks.ceph:[global] debug refs = 0/0 2023-11-08T12:11:07.846 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-11-08T12:11:07.846 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-11-08T12:11:07.846 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-11-08T12:11:07.847 INFO:tasks.ceph:[global] debug striper = 0/0 2023-11-08T12:11:07.847 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-11-08T12:11:07.847 INFO:tasks.ceph:[global] debug timer = 0/0 2023-11-08T12:11:07.848 INFO:tasks.ceph:[global] debug tp = 0/0 2023-11-08T12:11:07.848 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-08T12:11:07.848 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-08T12:11:07.848 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-08T12:11:07.849 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-08T12:11:07.849 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-08T12:11:07.849 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-08T12:11:07.849 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-08T12:11:07.850 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-08T12:11:07.850 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-08T12:11:07.850 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-11-08T12:11:07.850 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-11-08T12:11:07.851 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-11-08T12:11:07.851 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-08T12:11:07.851 INFO:tasks.ceph:Setting up mon.a... 2023-11-08T12:11:07.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-08T12:11:07.930 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.keyring 2023-11-08T12:11:07.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-08T12:11:07.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T12:11:08.018 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.49')] 2023-11-08T12:11:08.019 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.49', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-08T12:11:08.020 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:08.021 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T12:11:08.073 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.49 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:08.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:08.157+0000 7f2cf3a53e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:08.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:08.157+0000 7f2cf3a53e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:08.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:08.158+0000 7f2cf3a53e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid 7ba5e117-b189-41b9-97df-814e77e6b58c 2023-11-08T12:11:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = pacific 2023-11-08T12:11:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2023-11-08T12:11:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 7ba5e117-b189-41b9-97df-814e77e6b58c 2023-11-08T12:11:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2023-11-08T12:11:08.159720+0000 2023-11-08T12:11:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:created 2023-11-08T12:11:08.159720+0000 2023-11-08T12:11:08.163 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 16 (pacific) 2023-11-08T12:11:08.163 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 1 2023-11-08T12:11:08.163 INFO:teuthology.orchestra.run.smithi049.stdout:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2023-11-08T12:11:08.163 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-08T12:11:08.164 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T12:11:08.179 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7ba5e117-b189-41b9-97df-814e77e6b58c... 2023-11-08T12:11:08.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-08T12:11:08.279 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2023-11-08T12:11:08.280 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2023-11-08T12:11:08.280 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T12:11:08.280 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = none 2023-11-08T12:11:08.280 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.280 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2023-11-08T12:11:08.281 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.281 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2023-11-08T12:11:08.281 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.281 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2023-11-08T12:11:08.281 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2023-11-08T12:11:08.282 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.282 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2023-11-08T12:11:08.282 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2023-11-08T12:11:08.282 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.282 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T12:11:08.283 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2023-11-08T12:11:08.283 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.283 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T12:11:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T12:11:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2023-11-08T12:11:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2023-11-08T12:11:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2023-11-08T12:11:08.285 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2023-11-08T12:11:08.285 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2023-11-08T12:11:08.285 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2023-11-08T12:11:08.285 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2023-11-08T12:11:08.285 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2023-11-08T12:11:08.286 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T12:11:08.286 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T12:11:08.286 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2023-11-08T12:11:08.286 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.287 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-08T12:11:08.287 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.287 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2023-11-08T12:11:08.392 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2023-11-08T12:11:08.393 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2023-11-08T12:11:08.394 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2023-11-08T12:11:08.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2023-11-08T12:11:08.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = 172.21.15.49 2023-11-08T12:11:08.394 INFO:teuthology.orchestra.run.smithi049.stdout: auth client required = none 2023-11-08T12:11:08.395 INFO:teuthology.orchestra.run.smithi049.stdout: auth cluster required = none 2023-11-08T12:11:08.395 INFO:teuthology.orchestra.run.smithi049.stdout: auth service required = none 2023-11-08T12:11:08.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug asok = 0/0 2023-11-08T12:11:08.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 0/0 2023-11-08T12:11:08.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug bdev = 0/0 2023-11-08T12:11:08.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 0/0 2023-11-08T12:11:08.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 0/0 2023-11-08T12:11:08.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug buffer = 0/0 2023-11-08T12:11:08.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug civetweb = 0/0 2023-11-08T12:11:08.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug client = 0/0 2023-11-08T12:11:08.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug compressor = 0/0 2023-11-08T12:11:08.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug context = 0/0 2023-11-08T12:11:08.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug crush = 0/0 2023-11-08T12:11:08.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug crypto = 0/0 2023-11-08T12:11:08.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug dpdk = 0/0 2023-11-08T12:11:08.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug eventtrace = 0/0 2023-11-08T12:11:08.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug filer = 0/0 2023-11-08T12:11:08.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug filestore = 0/0 2023-11-08T12:11:08.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug finisher = 0/0 2023-11-08T12:11:08.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug fuse = 0/0 2023-11-08T12:11:08.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug heartbeatmap = 0/0 2023-11-08T12:11:08.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug javaclient = 0/0 2023-11-08T12:11:08.400 INFO:teuthology.orchestra.run.smithi049.stdout: debug journal = 0/0 2023-11-08T12:11:08.400 INFO:teuthology.orchestra.run.smithi049.stdout: debug journaler = 0/0 2023-11-08T12:11:08.400 INFO:teuthology.orchestra.run.smithi049.stdout: debug kstore = 0/0 2023-11-08T12:11:08.400 INFO:teuthology.orchestra.run.smithi049.stdout: debug leveldb = 0/0 2023-11-08T12:11:08.400 INFO:teuthology.orchestra.run.smithi049.stdout: debug lockdep = 0/0 2023-11-08T12:11:08.401 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds = 0/0 2023-11-08T12:11:08.401 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds balancer = 0/0 2023-11-08T12:11:08.401 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds locker = 0/0 2023-11-08T12:11:08.401 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds log = 0/0 2023-11-08T12:11:08.401 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds log expire = 0/0 2023-11-08T12:11:08.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds migrator = 0/0 2023-11-08T12:11:08.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug memdb = 0/0 2023-11-08T12:11:08.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 0/0 2023-11-08T12:11:08.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgrc = 0/0 2023-11-08T12:11:08.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 0/0 2023-11-08T12:11:08.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 0/0 2023-11-08T12:11:08.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2023-11-08T12:11:08.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug objclass = 0/0 2023-11-08T12:11:08.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug objectcacher = 0/0 2023-11-08T12:11:08.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug objecter = 0/0 2023-11-08T12:11:08.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug optracker = 0/0 2023-11-08T12:11:08.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 0/0 2023-11-08T12:11:08.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 0/0 2023-11-08T12:11:08.406 INFO:teuthology.orchestra.run.smithi049.stdout: debug perfcounter = 0/0 2023-11-08T12:11:08.406 INFO:teuthology.orchestra.run.smithi049.stdout: debug rados = 0/0 2023-11-08T12:11:08.406 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd = 0/0 2023-11-08T12:11:08.407 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd mirror = 0/0 2023-11-08T12:11:08.407 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd replay = 0/0 2023-11-08T12:11:08.407 INFO:teuthology.orchestra.run.smithi049.stdout: debug refs = 0/0 2023-11-08T12:11:08.408 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw = 0/0 2023-11-08T12:11:08.408 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw sync = 0/0 2023-11-08T12:11:08.408 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 0/0 2023-11-08T12:11:08.408 INFO:teuthology.orchestra.run.smithi049.stdout: debug striper = 0/0 2023-11-08T12:11:08.409 INFO:teuthology.orchestra.run.smithi049.stdout: debug throttle = 0/0 2023-11-08T12:11:08.409 INFO:teuthology.orchestra.run.smithi049.stdout: debug timer = 0/0 2023-11-08T12:11:08.409 INFO:teuthology.orchestra.run.smithi049.stdout: debug tp = 0/0 2023-11-08T12:11:08.409 INFO:teuthology.orchestra.run.smithi049.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-08T12:11:08.410 INFO:teuthology.orchestra.run.smithi049.stdout: ms client mode = crc 2023-11-08T12:11:08.410 INFO:teuthology.orchestra.run.smithi049.stdout: ms cluster mode = crc 2023-11-08T12:11:08.411 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon client mode = crc 2023-11-08T12:11:08.411 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon cluster mode = crc 2023-11-08T12:11:08.411 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon service mode = crc 2023-11-08T12:11:08.413 INFO:teuthology.orchestra.run.smithi049.stdout: ms service mode = crc 2023-11-08T12:11:08.414 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 7ba5e117-b189-41b9-97df-814e77e6b58c 2023-11-08T12:11:08.414 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.415 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2023-11-08T12:11:08.415 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2023-11-08T12:11:08.415 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.416 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2023-11-08T12:11:08.416 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2023-11-08T12:11:08.416 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2023-11-08T12:11:08.417 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.417 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2023-11-08T12:11:08.417 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2023-11-08T12:11:08.417 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.418 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2023-11-08T12:11:08.418 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2023-11-08T12:11:08.418 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2023-11-08T12:11:08.418 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug trim objects = true 2023-11-08T12:11:08.418 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.419 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2023-11-08T12:11:08.419 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2023-11-08T12:11:08.419 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.420 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2023-11-08T12:11:08.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.420 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2023-11-08T12:11:08.421 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.421 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2023-11-08T12:11:08.421 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.422 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2023-11-08T12:11:08.422 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2023-11-08T12:11:08.423 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.423 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2023-11-08T12:11:08.423 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2023-11-08T12:11:08.424 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 20 2023-11-08T12:11:08.424 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore block size = 96636764160 2023-11-08T12:11:08.424 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore compression algorithm = snappy 2023-11-08T12:11:08.425 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore compression mode = aggressive 2023-11-08T12:11:08.425 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore fsck on mount = True 2023-11-08T12:11:08.425 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore rocksdb cf = False 2023-11-08T12:11:08.426 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar smp = 3 2023-11-08T12:11:08.426 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 20 2023-11-08T12:11:08.427 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 20 2023-11-08T12:11:08.427 INFO:teuthology.orchestra.run.smithi049.stdout: debug filestore = 0/0 2023-11-08T12:11:08.427 INFO:teuthology.orchestra.run.smithi049.stdout: debug journal = 0/0 2023-11-08T12:11:08.428 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2023-11-08T12:11:08.428 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 0/0 2023-11-08T12:11:08.428 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 10 2023-11-08T12:11:08.429 INFO:teuthology.orchestra.run.smithi049.stdout: err to stderr = True 2023-11-08T12:11:08.429 INFO:teuthology.orchestra.run.smithi049.stdout: log flush on exit = True 2023-11-08T12:11:08.429 INFO:teuthology.orchestra.run.smithi049.stdout: log to file = False 2023-11-08T12:11:08.430 INFO:teuthology.orchestra.run.smithi049.stdout: log to stderr = True 2023-11-08T12:11:08.430 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T12:11:08.430 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd full ratio = 0.9 2023-11-08T12:11:08.431 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd nearfull ratio = 0.8 2023-11-08T12:11:08.431 INFO:teuthology.orchestra.run.smithi049.stdout: osd failsafe full ratio = 0.95 2023-11-08T12:11:08.431 INFO:teuthology.orchestra.run.smithi049.stdout: osd objectstore = bluestore 2023-11-08T12:11:08.432 INFO:teuthology.orchestra.run.smithi049.stdout: osd shutdown pgref assert = True 2023-11-08T12:11:08.432 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.432 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2023-11-08T12:11:08.433 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-11-08T12:11:08.433 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2023-11-08T12:11:08.433 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-11-08T12:11:08.434 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-11-08T12:11:08.434 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-11-08T12:11:08.435 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-11-08T12:11:08.435 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2023-11-08T12:11:08.435 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.436 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2023-11-08T12:11:08.436 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2023-11-08T12:11:08.436 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 0/0 2023-11-08T12:11:08.437 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 0/0 2023-11-08T12:11:08.437 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-11-08T12:11:08.437 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2023-11-08T12:11:08.438 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2023-11-08T12:11:08.438 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-11-08T12:11:08.438 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2023-11-08T12:11:08.439 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2023-11-08T12:11:08.439 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-11-08T12:11:08.439 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.440 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T12:11:08.440 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T12:11:08.441 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T12:11:08.441 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.441 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T12:11:08.442 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T12:11:08.442 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T12:11:08.442 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.443 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2023-11-08T12:11:08.443 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2023-11-08T12:11:08.443 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.444 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2023-11-08T12:11:08.444 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default crimson = True 2023-11-08T12:11:08.444 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:08.445 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2023-11-08T12:11:08.445 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2023-11-08T12:11:08.445 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2023-11-08T12:11:08.446 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2023-11-08T12:11:08.446 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T12:11:08.447 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T12:11:08.447 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2023-11-08T12:11:08.449 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-08T12:11:08.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-08T12:11:08.519 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-08T12:11:08.519 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:08.520 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-08T12:11:08.535 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:08.535 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-08T12:11:08.590 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:11:08.591 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:08.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-08T12:11:08.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T12:11:08.677 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:08.678 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:08.733 INFO:tasks.ceph:Setting up mon nodes... 2023-11-08T12:11:08.733 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-08T12:11:08.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-08T12:11:08.840 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-08T12:11:08.843 INFO:tasks.ceph:Setting up mds nodes... 2023-11-08T12:11:08.843 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-08T12:11:08.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-08T12:11:08.896 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-08T12:11:08.915 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-08T12:11:08.915 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T12:11:08.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-08T12:11:08.946 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:11:08.947 INFO:tasks.ceph:role: osd.0 2023-11-08T12:11:08.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:11:08.947 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-08T12:11:09.017 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T12:11:09.017 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T12:11:09.017 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T12:11:09.018 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-08T12:11:09.018 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T12:11:09.018 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-11-08T12:11:09.018 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T12:11:09.019 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-08T12:11:09.019 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T12:11:09.019 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T12:11:09.051 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-11-08T12:11:09.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-11-08T12:11:09.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-08T12:11:09.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-08T12:11:09.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-08T12:11:09.277 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:11:09.277 INFO:tasks.ceph:role: osd.1 2023-11-08T12:11:09.278 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:11:09.278 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-08T12:11:09.349 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T12:11:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T12:11:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T12:11:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-08T12:11:09.350 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T12:11:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-11-08T12:11:09.350 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T12:11:09.350 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-08T12:11:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T12:11:09.351 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T12:11:09.383 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-11-08T12:11:09.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-11-08T12:11:09.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-08T12:11:09.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-08T12:11:09.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-08T12:11:09.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:11:09.610 INFO:tasks.ceph:role: osd.2 2023-11-08T12:11:09.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:11:09.611 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-08T12:11:09.680 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T12:11:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T12:11:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T12:11:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-08T12:11:09.682 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T12:11:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-11-08T12:11:09.682 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T12:11:09.682 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-08T12:11:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T12:11:09.683 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T12:11:09.715 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-11-08T12:11:09.717 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-11-08T12:11:09.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-08T12:11:09.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-08T12:11:09.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:10.436 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:11:10.437 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:10,437 seastar - Reactor backend: linux-aio 2023-11-08T12:11:10.453 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:10,453 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:11:10.453 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:10,453 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:11:10.454 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:10,453 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:11:10.454 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:10,453 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:11:10.455 INFO:teuthology.orchestra.run.smithi049.stderr:==80041==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:11:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:==80041==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd6f078000; bottom 0x7f54b7aa7000; size: 0x00a8b75d1000 (724630835200) 2023-11-08T12:11:10.462 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-11-08T12:11:10.462 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:11:10.487 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xba189 0x2bc3e 0x2c061 0x2c0f0 0x2c3b8 0x2c5a7 0x2857c 0xb6323 0x22b1461b 0x22b12c6c 0x22ddafda 0x22ddd41e 0x22dddf49 0x22c3eba9 0x22d0d764 0x22d0f1af 0x22d0ff18 0x22cd7d84 0x22cd73dc 0x229e96c4 0x24a1bd30 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1ed04ea4 0x1ed0dc41 0x1eb80e12 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:11:10.488 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff84e7721c 0xffffffff8459e5ba 0xffffffff8459eed3 0xffffffff845a0acf 0xffffffff845a1e42 0xffffffff845a26b7 0xffffffff845c6e43 0xffffffff84572b53 0xffffffff8457ac6b 0xffffffff8457b08d 0xffffffff84289af4 0xffffffff84e65842 0xffffffff85000b82 2023-11-08T12:11:11.030 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:11.030 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-11-08T12:11:11.030 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-11-08T12:11:11.031 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-11-08T12:11:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:11.031 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:11.089 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:11:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:11,090 seastar - Reactor backend: linux-aio 2023-11-08T12:11:11.108 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:11,107 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:11:11.108 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:11,107 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:11:11.108 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:11,107 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:11:11.115 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:11,115 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:11:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:==80037==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:11:11.124 INFO:teuthology.orchestra.run.smithi049.stderr:==80037==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd6f078000; bottom 0x7f54b58a0000; size: 0x00a8b97d8000 (724666515456) 2023-11-08T12:11:11.124 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-11-08T12:11:11.124 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:11:11.150 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xb94d8 0x94469 0x94d46 0x22a1c81b 0x22b443ab 0x24a1c7fb 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:11:11.150 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2023-11-08T12:11:15.159 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:15.160 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-11-08T12:11:15.160 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-11-08T12:11:15.160 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-11-08T12:11:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:15.161 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:15.161 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 7ba5e117-b189-41b9-97df-814e77e6b58c 2023-11-08T12:11:15.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-08T12:11:15.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:15.781 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:11:15.782 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:15,782 seastar - Reactor backend: linux-aio 2023-11-08T12:11:15.798 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:15,798 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:11:15.798 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:15,798 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:11:15.799 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:15,798 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:11:15.799 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:15,798 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:11:15.801 INFO:teuthology.orchestra.run.smithi049.stderr:==80224==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:11:15.807 INFO:teuthology.orchestra.run.smithi049.stderr:==80224==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd54250000; bottom 0x7f580b251000; size: 0x00a548fff000 (709894336512) 2023-11-08T12:11:15.807 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-11-08T12:11:15.807 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:11:16.351 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-11-08T12:11:16.352 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-11-08T12:11:16.352 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-11-08T12:11:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:16.353 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:16.413 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:11:16.414 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:16,411 seastar - Reactor backend: linux-aio 2023-11-08T12:11:16.437 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:16,436 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:11:16.437 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:16,436 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:11:16.438 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:16,436 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:11:16.438 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:16,437 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:11:16.447 INFO:teuthology.orchestra.run.smithi049.stderr:==80222==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:11:16.453 INFO:teuthology.orchestra.run.smithi049.stderr:==80222==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd54250000; bottom 0x7f580904a000; size: 0x00a54b206000 (709930016768) 2023-11-08T12:11:16.454 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-11-08T12:11:16.454 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:11:16.479 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x94d4b 0x22a1fae1 0x22a1fe09 0x22aac4a7 0x22aaf6b7 0x22b398d9 0x22b3ea3f 0x22b440b0 0x24a1c7fb 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:11:16.479 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2023-11-08T12:11:20.493 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:20.493 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-11-08T12:11:20.493 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-11-08T12:11:20.494 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-11-08T12:11:20.494 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:20.494 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:20.494 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 7ba5e117-b189-41b9-97df-814e77e6b58c 2023-11-08T12:11:20.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-08T12:11:20.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:21.117 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:11:21.117 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,117 seastar - Reactor backend: linux-aio 2023-11-08T12:11:21.133 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,133 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:11:21.134 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,133 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:11:21.134 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,133 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:11:21.134 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,133 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:11:21.135 INFO:teuthology.orchestra.run.smithi049.stderr:==80406==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:11:21.142 INFO:teuthology.orchestra.run.smithi049.stderr:==80406==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffce1fa1000; bottom 0x7ff99497a000; size: 0x00034d627000 (14183198720) 2023-11-08T12:11:21.142 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-11-08T12:11:21.142 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:11:21.168 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x22cdc1a8 0x22cf4d57 0x22cf7ce8 0x22d1ca2c 0x22d3bca1 0x22d3c529 0x22cea0f9 0x22cf9719 0x22cfae15 0x22cfb7ee 0x22cfc8d9 0x22dde828 0x22c3eba9 0x22d0d764 0x22d0f1af 0x22d0ff18 0x22cd7d84 0x22cd73dc 0x229e96c4 0x24a1bd30 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1ed04ea4 0x1ed0dc41 0x1eb80e12 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:11:21.168 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2023-11-08T12:11:21.690 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:21.690 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-11-08T12:11:21.690 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-11-08T12:11:21.690 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-11-08T12:11:21.691 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:21.691 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:11:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,740 seastar - Reactor backend: linux-aio 2023-11-08T12:11:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,758 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:11:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,758 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:11:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,758 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:11:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-11-08 12:11:21,766 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:11:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:==80404==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:11:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:==80404==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffce1fa1000; bottom 0x7ff992773000; size: 0x00034f82e000 (14218878976) 2023-11-08T12:11:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-11-08T12:11:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:11:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x1e8c0cd0 0x22b3ed60 0x22b440b0 0x24a1c7fb 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:11:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2023-11-08T12:11:25.806 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:25.807 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-11-08T12:11:25.807 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-11-08T12:11:25.807 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-11-08T12:11:25.807 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-11-08T12:11:25.807 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:25.808 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 7ba5e117-b189-41b9-97df-814e77e6b58c 2023-11-08T12:11:25.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-08T12:11:25.877 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-08T12:11:25.878 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:25.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-08T12:11:25.948 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:25.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-08T12:11:26.018 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:26.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-08T12:11:26.087 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:26.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-08T12:11:26.158 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:26.158 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-08T12:11:26.213 INFO:tasks.ceph:Adding keys to all mons... 2023-11-08T12:11:26.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-08T12:11:26.282 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2023-11-08T12:11:26.283 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDcektl5UAVMhAAchNAClieMTXQEhy4iqlm1A== 2023-11-08T12:11:26.283 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2023-11-08T12:11:26.283 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDfektlpGpYChAAv2qwFhL6WeGtPdGCRqLvLA== 2023-11-08T12:11:26.283 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2023-11-08T12:11:26.284 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDkektlgEvxHRAAR/aKAhYAK+Qdy44hjLneXA== 2023-11-08T12:11:26.284 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2023-11-08T12:11:26.284 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDpektl+SkbMRAAqNBx48YQtj0/KIVBqCbieQ== 2023-11-08T12:11:26.284 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2023-11-08T12:11:26.284 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDcektlzu9rNRAAh5zb5lKCoW8ArnuSYkWIeQ== 2023-11-08T12:11:26.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T12:11:26.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T12:11:26.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T12:11:26.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T12:11:26.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T12:11:26.640 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-08T12:11:26.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-08T12:11:26.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-08T12:11:26.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:26.781+0000 7f028d849980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:26.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:26.781+0000 7f028d849980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:26.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-08T12:11:26.829 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:11:26.884 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-08T12:11:26.885 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-08T12:11:26.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-08T12:11:26.926 INFO:tasks.ceph.mon.a:Started 2023-11-08T12:11:26.927 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-08T12:11:26.927 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-08T12:11:26.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-08T12:11:26.930 INFO:tasks.ceph.mgr.x:Started 2023-11-08T12:11:26.930 INFO:tasks.ceph:Setting crush tunables to default 2023-11-08T12:11:26.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd crush tunables default 2023-11-08T12:11:27.012 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-11-08T12:11:27.011+0000 7f79ba645980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.013 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-11-08T12:11:27.011+0000 7f79ba645980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.013 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-11-08T12:11:27.012+0000 7f79ba645980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.016 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:27.015+0000 7fd068d6ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.053 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:27.047+0000 7fe727f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:27.070+0000 7fe727f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.141 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2023-11-08T12:11:27.151 INFO:tasks.ceph:check_enable_crimson: True 2023-11-08T12:11:27.152 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-08T12:11:27.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-08T12:11:27.219 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:27.218+0000 7fd068d6ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.220 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:27.218+0000 7fd068d6ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:27.257+0000 7f808782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:27.263+0000 7f808782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:27.363 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-11-08T12:11:27.362+0000 7f79b19f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-08T12:11:27.364 INFO:tasks.ceph.mon.a.smithi049.stderr:Please be aware that this feature is experimental, untested, 2023-11-08T12:11:27.364 INFO:tasks.ceph.mon.a.smithi049.stderr:unsupported, and may result in data corruption, data loss, 2023-11-08T12:11:27.364 INFO:tasks.ceph.mon.a.smithi049.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-08T12:11:27.364 INFO:tasks.ceph.mon.a.smithi049.stderr:feature with important data. 2023-11-08T12:11:27.365 INFO:tasks.ceph.mon.a.smithi049.stderr: 2023-11-08T12:11:27.697 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:27.696+0000 7fd068d6ff00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-08T12:11:27.751 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:27.750+0000 7fd068d6ff00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-08T12:11:28.153 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-08T12:11:28.154 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:28.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-08T12:11:28.185 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:28.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-08T12:11:28.255 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:28.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-08T12:11:28.309 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.308+0000 7fd068d6ff00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-08T12:11:28.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new c2ff436a-e1ec-4621-acbb-733e700f92e4 0 2023-11-08T12:11:28.364 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.363+0000 7fd068d6ff00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-08T12:11:28.412 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.411+0000 7fd068d6ff00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-08T12:11:28.470 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.469+0000 7fd068d6ff00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-08T12:11:28.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:28.469+0000 7f5ccb42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:28.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:28.476+0000 7f5ccb42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:28.515 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.514+0000 7fd068d6ff00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-08T12:11:28.555 INFO:teuthology.orchestra.run.smithi049.stdout:0 2023-11-08T12:11:28.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 91ddc7bb-14e6-4d24-b880-3d46ca1aaaa1 1 2023-11-08T12:11:28.622 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.621+0000 7fd068d6ff00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-08T12:11:28.677 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:28.671+0000 7fd5f4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:28.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:28.678+0000 7fd5f4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:28.698 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.697+0000 7fd068d6ff00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-08T12:11:28.746 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.745+0000 7fd068d6ff00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-08T12:11:28.756 INFO:teuthology.orchestra.run.smithi049.stdout:1 2023-11-08T12:11:28.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new bafd5119-7896-4f81-a5f0-4b75c9502783 2 2023-11-08T12:11:28.802 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.801+0000 7fd068d6ff00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-08T12:11:28.854 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:28.853+0000 7fd068d6ff00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-08T12:11:28.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:28.872+0000 7f8c96e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:28.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:28.880+0000 7f8c96e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:28.958 INFO:teuthology.orchestra.run.smithi049.stdout:2 2023-11-08T12:11:28.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-08T12:11:29.045 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-08T12:11:29.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-08T12:11:29.088 INFO:tasks.ceph.osd.0:Started 2023-11-08T12:11:29.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-08T12:11:29.124 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-08T12:11:29.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-08T12:11:29.167 INFO:tasks.ceph.osd.1:Started 2023-11-08T12:11:29.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-08T12:11:29.208 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-08T12:11:29.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-08T12:11:29.251 INFO:tasks.ceph.osd.2:Started 2023-11-08T12:11:29.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T12:11:29.273 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:29.272+0000 7fd068d6ff00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-08T12:11:29.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:29.454+0000 7fab056de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:29.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:29.466+0000 7fab056de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:29.515 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:29.514+0000 7fd068d6ff00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-08T12:11:29.605 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:29.605 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":6,"fsid":"7ba5e117-b189-41b9-97df-814e77e6b58c","created":"2023-11-08T12:11:27.061868+0000","modified":"2023-11-08T12:11:28.957513+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T12:11:28.957513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c2ff436a-e1ec-4621-acbb-733e700f92e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91ddc7bb-14e6-4d24-b880-3d46ca1aaaa1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bafd5119-7896-4f81-a5f0-4b75c9502783","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:11:29.627 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-08T12:11:29.627 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-08T12:11:29.685 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:29.684+0000 7fd068d6ff00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-08T12:11:29.737 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:29.736+0000 7fd068d6ff00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-08T12:11:29.842 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:29.841+0000 7fd068d6ff00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-08T12:11:29.930 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-08T12:11:29.947 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:29.946+0000 7fd068d6ff00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-08T12:11:30.034 INFO:teuthology.misc.health.smithi049.stderr:2023-11-08T12:11:30.030+0000 7fba20f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:30.037 INFO:teuthology.misc.health.smithi049.stderr:2023-11-08T12:11:30.036+0000 7fba20f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:30.105 INFO:teuthology.misc.health.smithi049.stdout: 2023-11-08T12:11:30.105 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":6,"fsid":"7ba5e117-b189-41b9-97df-814e77e6b58c","created":"2023-11-08T12:11:27.061868+0000","modified":"2023-11-08T12:11:28.957513+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T12:11:28.957513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c2ff436a-e1ec-4621-acbb-733e700f92e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91ddc7bb-14e6-4d24-b880-3d46ca1aaaa1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bafd5119-7896-4f81-a5f0-4b75c9502783","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:11:30.106 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:30.105+0000 7fd068d6ff00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-08T12:11:30.115 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-08T12:11:30.154 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:30.153+0000 7fd068d6ff00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-08T12:11:30.308 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:30.307+0000 7fd068d6ff00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-08T12:11:31.340 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:31.339+0000 7fd068d6ff00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-08T12:11:31.831 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:31.830+0000 7fd068d6ff00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-08T12:11:31.921 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:31.920+0000 7fd068d6ff00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-08T12:11:32.103 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:32.102+0000 7fd068d6ff00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-08T12:11:33.123 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-11-08T12:11:33.122+0000 7fd00899c640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-08T12:11:36.418 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-08T12:11:36.529 INFO:teuthology.misc.health.smithi049.stderr:2023-11-08T12:11:36.525+0000 7fae3855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:36.531 INFO:teuthology.misc.health.smithi049.stderr:2023-11-08T12:11:36.530+0000 7fae3855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:36.714 INFO:teuthology.misc.health.smithi049.stdout: 2023-11-08T12:11:36.715 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":8,"fsid":"7ba5e117-b189-41b9-97df-814e77e6b58c","created":"2023-11-08T12:11:27.061868+0000","modified":"2023-11-08T12:11:35.122837+0000","last_up_change":"2023-11-08T12:11:35.122837+0000","last_in_change":"2023-11-08T12:11:28.957513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c2ff436a-e1ec-4621-acbb-733e700f92e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":3407424988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":3407424988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":3407424988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3407424988}]},"public_addr":"172.21.15.49:6803/3407424988","cluster_addr":"172.21.15.49:6802/3407424988","heartbeat_back_addr":"172.21.15.49:6811/3407424988","heartbeat_front_addr":"172.21.15.49:6810/3407424988","state":["exists","up"]},{"osd":1,"uuid":"91ddc7bb-14e6-4d24-b880-3d46ca1aaaa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":685146040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":685146040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":685146040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":685146040}]},"public_addr":"172.21.15.49:6805/685146040","cluster_addr":"172.21.15.49:6804/685146040","heartbeat_back_addr":"172.21.15.49:6808/685146040","heartbeat_front_addr":"172.21.15.49:6809/685146040","state":["exists","up"]},{"osd":2,"uuid":"bafd5119-7896-4f81-a5f0-4b75c9502783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":17297941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":17297941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":17297941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":17297941}]},"public_addr":"172.21.15.49:6806/17297941","cluster_addr":"172.21.15.49:6807/17297941","heartbeat_back_addr":"172.21.15.49:6812/17297941","heartbeat_front_addr":"172.21.15.49:6813/17297941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:11:36.725 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-08T12:11:36.726 INFO:tasks.ceph:Creating RBD pool 2023-11-08T12:11:36.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-08T12:11:36.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:36.831+0000 7f260e00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:36.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:36.839+0000 7f260e00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:37.132 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2023-11-08T12:11:37.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-08T12:11:37.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:37.256+0000 7fba22b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:37.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:37.266+0000 7fba22b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:38.133 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-08T12:11:38.155 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-08T12:11:38.156 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-08T12:11:38.156 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-08T12:11:38.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-08T12:11:38.157 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-08T12:11:38.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:38.290+0000 7fe4e675e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:38.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:38.296+0000 7fe4e675e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:38.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":1549139948},{"type":"v1","addr":"172.21.15.49:6801","nonce":1549139948}]},"active_addr":"172.21.15.49:6801/1549139948","active_change":"2023-11-08T12:11:32.115268+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:v0.0.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2392015785}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1937287299}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1328096045}]}]} 2023-11-08T12:11:38.550 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-08T12:11:38.550 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-08T12:11:38.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T12:11:38.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:38.663+0000 7fdcba35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:38.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:38.669+0000 7fdcba35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:38.980 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:38.980 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":10,"fsid":"7ba5e117-b189-41b9-97df-814e77e6b58c","created":"2023-11-08T12:11:27.061868+0000","modified":"2023-11-08T12:11:38.132096+0000","last_up_change":"2023-11-08T12:11:35.122837+0000","last_in_change":"2023-11-08T12:11:28.957513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-08T12:11:36.172077+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T12:11:37.053737+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2ff436a-e1ec-4621-acbb-733e700f92e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":3407424988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":3407424988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":3407424988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3407424988}]},"public_addr":"172.21.15.49:6803/3407424988","cluster_addr":"172.21.15.49:6802/3407424988","heartbeat_back_addr":"172.21.15.49:6811/3407424988","heartbeat_front_addr":"172.21.15.49:6810/3407424988","state":["exists","up"]},{"osd":1,"uuid":"91ddc7bb-14e6-4d24-b880-3d46ca1aaaa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":685146040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":685146040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":685146040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":685146040}]},"public_addr":"172.21.15.49:6805/685146040","cluster_addr":"172.21.15.49:6804/685146040","heartbeat_back_addr":"172.21.15.49:6808/685146040","heartbeat_front_addr":"172.21.15.49:6809/685146040","state":["exists","up"]},{"osd":2,"uuid":"bafd5119-7896-4f81-a5f0-4b75c9502783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":17297941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":17297941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":17297941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":17297941}]},"public_addr":"172.21.15.49:6806/17297941","cluster_addr":"172.21.15.49:6807/17297941","heartbeat_back_addr":"172.21.15.49:6812/17297941","heartbeat_front_addr":"172.21.15.49:6813/17297941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:11:38.993 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-08T12:11:38.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T12:11:39.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.113+0000 7f12f2b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.119+0000 7f12f2b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.311 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:39.311 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"fsid":"7ba5e117-b189-41b9-97df-814e77e6b58c","created":"2023-11-08T12:11:27.061868+0000","modified":"2023-11-08T12:11:39.140686+0000","last_up_change":"2023-11-08T12:11:35.122837+0000","last_in_change":"2023-11-08T12:11:28.957513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-08T12:11:36.172077+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T12:11:37.053737+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2ff436a-e1ec-4621-acbb-733e700f92e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":3407424988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":3407424988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":3407424988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3407424988}]},"public_addr":"172.21.15.49:6803/3407424988","cluster_addr":"172.21.15.49:6802/3407424988","heartbeat_back_addr":"172.21.15.49:6811/3407424988","heartbeat_front_addr":"172.21.15.49:6810/3407424988","state":["exists","up"]},{"osd":1,"uuid":"91ddc7bb-14e6-4d24-b880-3d46ca1aaaa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":685146040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":685146040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":685146040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":685146040}]},"public_addr":"172.21.15.49:6805/685146040","cluster_addr":"172.21.15.49:6804/685146040","heartbeat_back_addr":"172.21.15.49:6808/685146040","heartbeat_front_addr":"172.21.15.49:6809/685146040","state":["exists","up"]},{"osd":2,"uuid":"bafd5119-7896-4f81-a5f0-4b75c9502783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":17297941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":17297941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":17297941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":17297941}]},"public_addr":"172.21.15.49:6806/17297941","cluster_addr":"172.21.15.49:6807/17297941","heartbeat_back_addr":"172.21.15.49:6812/17297941","heartbeat_front_addr":"172.21.15.49:6813/17297941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:11:39.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-08T12:11:39.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-08T12:11:39.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-08T12:11:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.439+0000 7f9920cf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.439+0000 7f8389515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.445 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.439+0000 7f537d79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.446+0000 7f537d79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.446+0000 7f8389515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.446+0000 7f9920cf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.472 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2023-11-08T12:11:39.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-08T12:11:39.475 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2023-11-08T12:11:39.475 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2023-11-08T12:11:39.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-08T12:11:39.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-08T12:11:39.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.583+0000 7f2a4899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.589+0000 7f2a4899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.633 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.629+0000 7fa44c19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.629+0000 7f3636b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.635+0000 7f3636b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.635+0000 7fa44c19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.781 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2023-11-08T12:11:39.792 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-11-08T12:11:39.793 DEBUG:teuthology.parallel:result is None 2023-11-08T12:11:39.826 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2023-11-08T12:11:39.827 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2023-11-08T12:11:39.839 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-11-08T12:11:39.839 DEBUG:teuthology.parallel:result is None 2023-11-08T12:11:39.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-11-08T12:11:39.841 DEBUG:teuthology.parallel:result is None 2023-11-08T12:11:39.841 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-08T12:11:39.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T12:11:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.988+0000 7fee6211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:39.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:39.993+0000 7fee6211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:40.178 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:40.179 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2023-11-08T12:11:40.190 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-11-08T12:11:40.119652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80768,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034752,"statfs":{"total":289910292480,"available":289827586048,"internally_reserved":0,"allocated":262144,"data_stored":1221228,"data_compressed":59690,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.657449"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.152434+0000","last_change":"2023-11-08T12:11:38.273179+0000","last_active":"2023-11-08T12:11:39.152434+0000","last_peered":"2023-11-08T12:11:39.152434+0000","last_clean":"2023-11-08T12:11:39.152434+0000","last_became_active":"2023-11-08T12:11:38.270972+0000","last_became_peered":"2023-11-08T12:11:38.270972+0000","last_unstale":"2023-11-08T12:11:39.152434+0000","last_undegraded":"2023-11-08T12:11:39.152434+0000","last_fullsized":"2023-11-08T12:11:39.152434+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.149369+0000","last_change":"2023-11-08T12:11:38.284040+0000","last_active":"2023-11-08T12:11:39.149369+0000","last_peered":"2023-11-08T12:11:39.149369+0000","last_clean":"2023-11-08T12:11:39.149369+0000","last_became_active":"2023-11-08T12:11:38.275647+0000","last_became_peered":"2023-11-08T12:11:38.275647+0000","last_unstale":"2023-11-08T12:11:39.149369+0000","last_undegraded":"2023-11-08T12:11:39.149369+0000","last_fullsized":"2023-11-08T12:11:39.149369+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.151011+0000","last_change":"2023-11-08T12:11:38.257816+0000","last_active":"2023-11-08T12:11:39.151011+0000","last_peered":"2023-11-08T12:11:39.151011+0000","last_clean":"2023-11-08T12:11:39.151011+0000","last_became_active":"2023-11-08T12:11:38.254555+0000","last_became_peered":"2023-11-08T12:11:38.254555+0000","last_unstale":"2023-11-08T12:11:39.151011+0000","last_undegraded":"2023-11-08T12:11:39.151011+0000","last_fullsized":"2023-11-08T12:11:39.151011+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.150810+0000","last_change":"2023-11-08T12:11:38.290774+0000","last_active":"2023-11-08T12:11:39.150810+0000","last_peered":"2023-11-08T12:11:39.150810+0000","last_clean":"2023-11-08T12:11:39.150810+0000","last_became_active":"2023-11-08T12:11:38.289024+0000","last_became_peered":"2023-11-08T12:11:38.289024+0000","last_unstale":"2023-11-08T12:11:39.150810+0000","last_undegraded":"2023-11-08T12:11:39.150810+0000","last_fullsized":"2023-11-08T12:11:39.150810+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T12:11:38.243506+0000","last_change":"2023-11-08T12:11:38.243626+0000","last_active":"2023-11-08T12:11:38.243506+0000","last_peered":"2023-11-08T12:11:38.243506+0000","last_clean":"2023-11-08T12:11:37.129583+0000","last_became_active":"2023-11-08T12:11:38.241334+0000","last_became_peered":"2023-11-08T12:11:38.241334+0000","last_unstale":"2023-11-08T12:11:38.243506+0000","last_undegraded":"2023-11-08T12:11:38.243506+0000","last_fullsized":"2023-11-08T12:11:38.243506+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T12:11:38.235966+0000","last_change":"2023-11-08T12:11:38.236024+0000","last_active":"2023-11-08T12:11:38.235966+0000","last_peered":"2023-11-08T12:11:38.235966+0000","last_clean":"2023-11-08T12:11:37.129583+0000","last_became_active":"2023-11-08T12:11:38.234091+0000","last_became_peered":"2023-11-08T12:11:38.234091+0000","last_unstale":"2023-11-08T12:11:38.235966+0000","last_undegraded":"2023-11-08T12:11:38.235966+0000","last_fullsized":"2023-11-08T12:11:38.235966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T12:11:38.232612+0000","last_change":"2023-11-08T12:11:38.232726+0000","last_active":"2023-11-08T12:11:38.232612+0000","last_peered":"2023-11-08T12:11:38.232612+0000","last_clean":"2023-11-08T12:11:37.129583+0000","last_became_active":"2023-11-08T12:11:38.229964+0000","last_became_peered":"2023-11-08T12:11:38.229964+0000","last_unstale":"2023-11-08T12:11:38.232612+0000","last_undegraded":"2023-11-08T12:11:38.232612+0000","last_fullsized":"2023-11-08T12:11:38.232612+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.149378+0000","last_change":"2023-11-08T12:11:38.224764+0000","last_active":"2023-11-08T12:11:39.149378+0000","last_peered":"2023-11-08T12:11:39.149378+0000","last_clean":"2023-11-08T12:11:39.149378+0000","last_became_active":"2023-11-08T12:11:38.222617+0000","last_became_peered":"2023-11-08T12:11:38.222617+0000","last_unstale":"2023-11-08T12:11:39.149378+0000","last_undegraded":"2023-11-08T12:11:39.149378+0000","last_fullsized":"2023-11-08T12:11:39.149378+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.230563+0000","last_change":"2023-11-08T12:11:38.249722+0000","last_active":"2023-11-08T12:11:39.230563+0000","last_peered":"2023-11-08T12:11:39.230563+0000","last_clean":"2023-11-08T12:11:39.230563+0000","last_became_active":"2023-11-08T12:11:38.246454+0000","last_became_peered":"2023-11-08T12:11:38.246454+0000","last_unstale":"2023-11-08T12:11:39.230563+0000","last_undegraded":"2023-11-08T12:11:39.230563+0000","last_fullsized":"2023-11-08T12:11:39.230563+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26868,"kb_used_data":44,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344972,"statfs":{"total":96636764160,"available":96609251328,"internally_reserved":0,"allocated":45056,"data_stored":12727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":110592,"data_stored":605406,"data_compressed":29845,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":106496,"data_stored":603095,"data_compressed":29845,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-08T12:11:40.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T12:11:40.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:40.300+0000 7fd327806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:40.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:40.305+0000 7fd327806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:40.485 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2023-11-08T12:11:40.496 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-11-08T12:11:40.119652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80768,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034752,"statfs":{"total":289910292480,"available":289827586048,"internally_reserved":0,"allocated":262144,"data_stored":1221228,"data_compressed":59690,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.657449"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.152434+0000","last_change":"2023-11-08T12:11:38.273179+0000","last_active":"2023-11-08T12:11:39.152434+0000","last_peered":"2023-11-08T12:11:39.152434+0000","last_clean":"2023-11-08T12:11:39.152434+0000","last_became_active":"2023-11-08T12:11:38.270972+0000","last_became_peered":"2023-11-08T12:11:38.270972+0000","last_unstale":"2023-11-08T12:11:39.152434+0000","last_undegraded":"2023-11-08T12:11:39.152434+0000","last_fullsized":"2023-11-08T12:11:39.152434+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.149369+0000","last_change":"2023-11-08T12:11:38.284040+0000","last_active":"2023-11-08T12:11:39.149369+0000","last_peered":"2023-11-08T12:11:39.149369+0000","last_clean":"2023-11-08T12:11:39.149369+0000","last_became_active":"2023-11-08T12:11:38.275647+0000","last_became_peered":"2023-11-08T12:11:38.275647+0000","last_unstale":"2023-11-08T12:11:39.149369+0000","last_undegraded":"2023-11-08T12:11:39.149369+0000","last_fullsized":"2023-11-08T12:11:39.149369+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.151011+0000","last_change":"2023-11-08T12:11:38.257816+0000","last_active":"2023-11-08T12:11:39.151011+0000","last_peered":"2023-11-08T12:11:39.151011+0000","last_clean":"2023-11-08T12:11:39.151011+0000","last_became_active":"2023-11-08T12:11:38.254555+0000","last_became_peered":"2023-11-08T12:11:38.254555+0000","last_unstale":"2023-11-08T12:11:39.151011+0000","last_undegraded":"2023-11-08T12:11:39.151011+0000","last_fullsized":"2023-11-08T12:11:39.151011+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.150810+0000","last_change":"2023-11-08T12:11:38.290774+0000","last_active":"2023-11-08T12:11:39.150810+0000","last_peered":"2023-11-08T12:11:39.150810+0000","last_clean":"2023-11-08T12:11:39.150810+0000","last_became_active":"2023-11-08T12:11:38.289024+0000","last_became_peered":"2023-11-08T12:11:38.289024+0000","last_unstale":"2023-11-08T12:11:39.150810+0000","last_undegraded":"2023-11-08T12:11:39.150810+0000","last_fullsized":"2023-11-08T12:11:39.150810+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T12:11:38.243506+0000","last_change":"2023-11-08T12:11:38.243626+0000","last_active":"2023-11-08T12:11:38.243506+0000","last_peered":"2023-11-08T12:11:38.243506+0000","last_clean":"2023-11-08T12:11:37.129583+0000","last_became_active":"2023-11-08T12:11:38.241334+0000","last_became_peered":"2023-11-08T12:11:38.241334+0000","last_unstale":"2023-11-08T12:11:38.243506+0000","last_undegraded":"2023-11-08T12:11:38.243506+0000","last_fullsized":"2023-11-08T12:11:38.243506+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T12:11:38.235966+0000","last_change":"2023-11-08T12:11:38.236024+0000","last_active":"2023-11-08T12:11:38.235966+0000","last_peered":"2023-11-08T12:11:38.235966+0000","last_clean":"2023-11-08T12:11:37.129583+0000","last_became_active":"2023-11-08T12:11:38.234091+0000","last_became_peered":"2023-11-08T12:11:38.234091+0000","last_unstale":"2023-11-08T12:11:38.235966+0000","last_undegraded":"2023-11-08T12:11:38.235966+0000","last_fullsized":"2023-11-08T12:11:38.235966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T12:11:38.232612+0000","last_change":"2023-11-08T12:11:38.232726+0000","last_active":"2023-11-08T12:11:38.232612+0000","last_peered":"2023-11-08T12:11:38.232612+0000","last_clean":"2023-11-08T12:11:37.129583+0000","last_became_active":"2023-11-08T12:11:38.229964+0000","last_became_peered":"2023-11-08T12:11:38.229964+0000","last_unstale":"2023-11-08T12:11:38.232612+0000","last_undegraded":"2023-11-08T12:11:38.232612+0000","last_fullsized":"2023-11-08T12:11:38.232612+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.149378+0000","last_change":"2023-11-08T12:11:38.224764+0000","last_active":"2023-11-08T12:11:39.149378+0000","last_peered":"2023-11-08T12:11:39.149378+0000","last_clean":"2023-11-08T12:11:39.149378+0000","last_became_active":"2023-11-08T12:11:38.222617+0000","last_became_peered":"2023-11-08T12:11:38.222617+0000","last_unstale":"2023-11-08T12:11:39.149378+0000","last_undegraded":"2023-11-08T12:11:39.149378+0000","last_fullsized":"2023-11-08T12:11:39.149378+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:11:39.230563+0000","last_change":"2023-11-08T12:11:38.249722+0000","last_active":"2023-11-08T12:11:39.230563+0000","last_peered":"2023-11-08T12:11:39.230563+0000","last_clean":"2023-11-08T12:11:39.230563+0000","last_became_active":"2023-11-08T12:11:38.246454+0000","last_became_peered":"2023-11-08T12:11:38.246454+0000","last_unstale":"2023-11-08T12:11:39.230563+0000","last_undegraded":"2023-11-08T12:11:39.230563+0000","last_fullsized":"2023-11-08T12:11:39.230563+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:11:37.129583+0000","last_clean_scrub_stamp":"2023-11-08T12:11:37.129583+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26868,"kb_used_data":44,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344972,"statfs":{"total":96636764160,"available":96609251328,"internally_reserved":0,"allocated":45056,"data_stored":12727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":110592,"data_stored":605406,"data_compressed":29845,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":106496,"data_stored":603095,"data_compressed":29845,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-08T12:11:40.497 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-08T12:11:40.498 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-08T12:11:40.498 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-08T12:11:40.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-08T12:11:40.610 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:40.607+0000 7f71be6f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:40.612 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:11:40.611+0000 7f71be6f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:11:40.837 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:40.838 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-08T12:11:40.849 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-08T12:11:40.850 INFO:teuthology.run_tasks:Running task cbt... 2023-11-08T12:11:40.866 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi049.front.sepia.ceph.com', 'osds': ['ubuntu@smithi049.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi049.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi049.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi049.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-11-08T12:11:40.867 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-11-08T12:11:40.885 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:11:40.885 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-08T12:11:40.937 DEBUG:teuthology.orchestra.run.smithi049:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-11-08T12:11:40.991 INFO:teuthology.orchestra.run.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-11-08T12:11:41.516 INFO:tasks.cbt:Installing collectl 2023-11-08T12:11:41.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-11-08T12:11:41.635 INFO:teuthology.orchestra.run.smithi049.stderr:--2023-11-08 12:11:41-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-11-08T12:11:41.788 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2023-11-08T12:11:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-11-08T12:11:42.044 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-08T12:11:42.044 INFO:teuthology.orchestra.run.smithi049.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlS3r-QEkHIUtJj8jnx9dE8S77NhCouckl8raOln7tXxfe8TrcwnLK4hBQdS1ymN1ZaE0gy6U5BUGZyIpRpS1Y7pbqDQ%3D%3D&use_mirror=versaweb&r= [following] 2023-11-08T12:11:42.044 INFO:teuthology.orchestra.run.smithi049.stderr:--2023-11-08 12:11:42-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlS3r-QEkHIUtJj8jnx9dE8S77NhCouckl8raOln7tXxfe8TrcwnLK4hBQdS1ymN1ZaE0gy6U5BUGZyIpRpS1Y7pbqDQ%3D%3D&use_mirror=versaweb&r= 2023-11-08T12:11:42.044 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-11-08T12:11:42.115 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-11-08T12:11:42.349 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-08T12:11:42.350 INFO:teuthology.orchestra.run.smithi049.stderr:Location: https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-11-08T12:11:42.350 INFO:teuthology.orchestra.run.smithi049.stderr:--2023-11-08 12:11:42-- https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-11-08T12:11:42.360 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)... 162.251.232.173 2023-11-08T12:11:42.431 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)|162.251.232.173|:443... connected. 2023-11-08T12:11:42.723 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 200 OK 2023-11-08T12:11:42.724 INFO:teuthology.orchestra.run.smithi049.stderr:Length: 620693 (606K) [application/x-gzip] 2023-11-08T12:11:42.724 INFO:teuthology.orchestra.run.smithi049.stderr:Saving to: ‘collectl.tar.gz’ 2023-11-08T12:11:42.724 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:42.794 INFO:teuthology.orchestra.run.smithi049.stderr: 0K .......... .......... .......... .......... .......... 8% 711K 1s 2023-11-08T12:11:42.864 INFO:teuthology.orchestra.run.smithi049.stderr: 50K .......... .......... .......... .......... .......... 16% 714K 1s 2023-11-08T12:11:42.864 INFO:teuthology.orchestra.run.smithi049.stderr: 100K .......... .......... .......... .......... .......... 24% 60.4M 0s 2023-11-08T12:11:42.933 INFO:teuthology.orchestra.run.smithi049.stderr: 150K .......... .......... .......... .......... .......... 32% 729K 0s 2023-11-08T12:11:42.934 INFO:teuthology.orchestra.run.smithi049.stderr: 200K .......... .......... .......... .......... .......... 41% 55.4M 0s 2023-11-08T12:11:42.935 INFO:teuthology.orchestra.run.smithi049.stderr: 250K .......... .......... .......... .......... .......... 49% 28.7M 0s 2023-11-08T12:11:42.937 INFO:teuthology.orchestra.run.smithi049.stderr: 300K .......... .......... .......... .......... .......... 57% 29.9M 0s 2023-11-08T12:11:43.004 INFO:teuthology.orchestra.run.smithi049.stderr: 350K .......... .......... .......... .......... .......... 65% 761K 0s 2023-11-08T12:11:43.004 INFO:teuthology.orchestra.run.smithi049.stderr: 400K .......... .......... .......... .......... .......... 74% 144M 0s 2023-11-08T12:11:43.004 INFO:teuthology.orchestra.run.smithi049.stderr: 450K .......... .......... .......... .......... .......... 82% 51.2M 0s 2023-11-08T12:11:43.005 INFO:teuthology.orchestra.run.smithi049.stderr: 500K .......... .......... .......... .......... .......... 90% 49.2M 0s 2023-11-08T12:11:43.007 INFO:teuthology.orchestra.run.smithi049.stderr: 550K .......... .......... .......... .......... .......... 98% 31.2M 0s 2023-11-08T12:11:43.007 INFO:teuthology.orchestra.run.smithi049.stderr: 600K ...... 100% 11.4T=0.3s 2023-11-08T12:11:43.007 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:43.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08 12:11:43 (2.09 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-11-08T12:11:43.008 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:11:43.046 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/ 2023-11-08T12:11:43.047 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/collectl.conf 2023-11-08T12:11:43.047 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ 2023-11-08T12:11:43.047 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/SlowProc.html 2023-11-08T12:11:43.047 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-11-08T12:11:43.047 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/WhySummary.html 2023-11-08T12:11:43.048 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Socket.html 2023-11-08T12:11:43.048 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/colmux.html 2023-11-08T12:11:43.048 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Tutorial.html 2023-11-08T12:11:43.048 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Gexpr.html 2023-11-08T12:11:43.048 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Import.html 2023-11-08T12:11:43.049 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/style.css 2023-11-08T12:11:43.049 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-11-08T12:11:43.049 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-11-08T12:11:43.049 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/FileNaming.html 2023-11-08T12:11:43.049 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/TheMath.html 2023-11-08T12:11:43.050 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Environmental.html 2023-11-08T12:11:43.050 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Process.html 2023-11-08T12:11:43.050 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-11-08T12:11:43.050 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/HiResTime.html 2023-11-08T12:11:43.051 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-11-08T12:11:43.051 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/slub.jpg 2023-11-08T12:11:43.051 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Lustre.html 2023-11-08T12:11:43.051 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Documentation.html 2023-11-08T12:11:43.051 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Inodes.html 2023-11-08T12:11:43.052 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Graphite.html 2023-11-08T12:11:43.052 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/InputFiles.html 2023-11-08T12:11:43.052 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Export.html 2023-11-08T12:11:43.052 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Interrupts.html 2023-11-08T12:11:43.052 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Startup.html 2023-11-08T12:11:43.053 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Performance.html 2023-11-08T12:11:43.053 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Messages.html 2023-11-08T12:11:43.053 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Logging.html 2023-11-08T12:11:43.053 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-detail.html 2023-11-08T12:11:43.054 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/CPUs.html 2023-11-08T12:11:43.054 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Matrix.html 2023-11-08T12:11:43.054 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-brief.html 2023-11-08T12:11:43.054 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-11-08T12:11:43.054 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Misc.html 2023-11-08T12:11:43.055 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Examples.html 2023-11-08T12:11:43.055 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OpenStack.html 2023-11-08T12:11:43.055 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-11-08T12:11:43.055 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-11-08T12:11:43.055 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-11-08T12:11:43.056 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-11-08T12:11:43.056 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data.html 2023-11-08T12:11:43.056 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Exceptions.html 2023-11-08T12:11:43.056 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-11-08T12:11:43.057 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-11-08T12:11:43.057 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-11-08T12:11:43.057 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-11-08T12:11:43.057 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Playback.html 2023-11-08T12:11:43.057 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Hello.html 2023-11-08T12:11:43.058 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Sockets.html 2023-11-08T12:11:43.058 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Network.html 2023-11-08T12:11:43.058 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Infiniband.html 2023-11-08T12:11:43.058 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Memory.html 2023-11-08T12:11:43.058 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-11-08T12:11:43.059 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Disks.html 2023-11-08T12:11:43.059 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-11-08T12:11:43.059 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/service/ 2023-11-08T12:11:43.059 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/service/collectl.service 2023-11-08T12:11:43.059 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/ 2023-11-08T12:11:43.060 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-suse 2023-11-08T12:11:43.060 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-debian 2023-11-08T12:11:43.060 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-generic 2023-11-08T12:11:43.060 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl 2023-11-08T12:11:43.060 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/misc.ph 2023-11-08T12:11:43.061 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/gexpr.ph 2023-11-08T12:11:43.061 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/statsd.ph 2023-11-08T12:11:43.061 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/envrules.std 2023-11-08T12:11:43.061 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/hello.ph 2023-11-08T12:11:43.062 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vmsum.ph 2023-11-08T12:11:43.062 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/proctree.ph 2023-11-08T12:11:43.062 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/README-WINDOWS 2023-11-08T12:11:43.062 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/formatit.ph 2023-11-08T12:11:43.064 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/colmux 2023-11-08T12:11:43.065 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/INSTALL 2023-11-08T12:11:43.065 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/graphite.ph 2023-11-08T12:11:43.065 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/RELEASE-collectl 2023-11-08T12:11:43.065 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vmstat.ph 2023-11-08T12:11:43.065 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/README 2023-11-08T12:11:43.066 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/ 2023-11-08T12:11:43.066 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/collectl.1 2023-11-08T12:11:43.066 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/colmux.1 2023-11-08T12:11:43.066 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/lexpr.ph 2023-11-08T12:11:43.067 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/client.pl 2023-11-08T12:11:43.067 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/collectl 2023-11-08T12:11:43.067 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/COPYING 2023-11-08T12:11:43.067 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vnet.ph 2023-11-08T12:11:43.067 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/GPL 2023-11-08T12:11:43.068 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/UNINSTALL 2023-11-08T12:11:43.068 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/ARTISTIC 2023-11-08T12:11:43.112 INFO:teuthology.orchestra.run.smithi049.stderr:./INSTALL: line 90: chkconfig: command not found 2023-11-08T12:11:43.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-11-08T12:11:43.717 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:23 ago on Wed 08 Nov 2023 12:08:20 PM UTC. 2023-11-08T12:11:43.816 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-11-08T12:11:43.818 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-11-08T12:11:43.878 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-08T12:11:43.878 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:43.878 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-08T12:11:43.878 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:43.879 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-08T12:11:43.879 INFO:teuthology.orchestra.run.smithi049.stdout: librbd-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 23 k 2023-11-08T12:11:43.879 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-11-08T12:11:43.879 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-11-08T12:11:43.879 INFO:teuthology.orchestra.run.smithi049.stdout: perf x86_64 5.14.0-382.el9 CentOS-AppStream 7.4 M 2023-11-08T12:11:43.879 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-08T12:11:43.880 INFO:teuthology.orchestra.run.smithi049.stdout: libradospp-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 30 k 2023-11-08T12:11:43.880 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:43.880 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-08T12:11:43.880 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-08T12:11:43.880 INFO:teuthology.orchestra.run.smithi049.stdout:Install 5 Packages 2023-11-08T12:11:43.881 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:43.881 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 7.6 M 2023-11-08T12:11:43.881 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 11 M 2023-11-08T12:11:43.882 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-08T12:11:44.152 INFO:teuthology.orchestra.run.smithi049.stdout:(1/5): librbd-devel-18.0.0-7227.g25e18238.el9.x 125 kB/s | 23 kB 00:00 2023-11-08T12:11:44.244 INFO:teuthology.orchestra.run.smithi049.stdout:(2/5): perf-5.14.0-382.el9.x86_64.rpm 27 MB/s | 7.4 MB 00:00 2023-11-08T12:11:44.286 INFO:teuthology.orchestra.run.smithi049.stdout:(3/5): libradospp-devel-18.0.0-7227.g25e18238.e 94 kB/s | 30 kB 00:00 2023-11-08T12:11:44.478 INFO:teuthology.orchestra.run.smithi049.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 56 kB/s | 13 kB 00:00 2023-11-08T12:11:44.612 INFO:teuthology.orchestra.run.smithi049.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 260 kB/s | 118 kB 00:00 2023-11-08T12:11:44.613 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:11:44.614 INFO:teuthology.orchestra.run.smithi049.stdout:Total 10 MB/s | 7.6 MB 00:00 2023-11-08T12:11:44.614 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-08T12:11:44.637 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-08T12:11:44.637 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-08T12:11:44.810 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-08T12:11:44.811 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-08T12:11:45.248 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-08T12:11:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-11-08T12:11:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-11-08T12:11:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradospp-devel-2:18.0.0-7227.g25e18238.el9.x86_64 3/5 2023-11-08T12:11:45.739 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librbd-devel-2:18.0.0-7227.g25e18238.el9.x86_64 4/5 2023-11-08T12:11:45.875 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : perf-5.14.0-382.el9.x86_64 5/5 2023-11-08T12:11:46.844 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: perf-5.14.0-382.el9.x86_64 5/5 2023-11-08T12:11:46.845 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradospp-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/5 2023-11-08T12:11:46.845 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2/5 2023-11-08T12:11:46.845 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : perf-5.14.0-382.el9.x86_64 3/5 2023-11-08T12:11:46.845 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-11-08T12:11:47.206 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-11-08T12:11:47.207 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:47.207 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-08T12:11:47.207 INFO:teuthology.orchestra.run.smithi049.stdout: libradospp-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:11:47.207 INFO:teuthology.orchestra.run.smithi049.stdout: librbd-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:11:47.207 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-2.34-7.el9.x86_64 2023-11-08T12:11:47.208 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-11-08T12:11:47.208 INFO:teuthology.orchestra.run.smithi049.stdout: perf-5.14.0-382.el9.x86_64 2023-11-08T12:11:47.208 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:47.208 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-08T12:11:47.596 INFO:tasks.cbt:benchmark: librbdfio 2023-11-08T12:11:47.597 DEBUG:teuthology.orchestra.run.smithi049:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2023-11-08T12:11:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2023-11-08T12:11:49.725 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest/fio && ./configure && make 2023-11-08T12:11:50.298 INFO:teuthology.orchestra.run.smithi049.stdout:Operating system Linux 2023-11-08T12:11:50.299 INFO:teuthology.orchestra.run.smithi049.stdout:CPU x86_64 2023-11-08T12:11:50.299 INFO:teuthology.orchestra.run.smithi049.stdout:Big endian no 2023-11-08T12:11:50.299 INFO:teuthology.orchestra.run.smithi049.stdout:Compiler gcc 2023-11-08T12:11:50.299 INFO:teuthology.orchestra.run.smithi049.stdout:Cross compile no 2023-11-08T12:11:50.299 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:11:50.300 INFO:teuthology.orchestra.run.smithi049.stdout:Static build no 2023-11-08T12:11:50.415 INFO:teuthology.orchestra.run.smithi049.stdout:Wordsize 64 2023-11-08T12:11:50.422 INFO:teuthology.orchestra.run.smithi049.stdout:zlib no 2023-11-08T12:11:50.468 INFO:teuthology.orchestra.run.smithi049.stdout:fcntl(F_FULLFSYNC) no 2023-11-08T12:11:50.481 INFO:teuthology.orchestra.run.smithi049.stdout:Linux AIO support no 2023-11-08T12:11:50.482 INFO:teuthology.orchestra.run.smithi049.stdout:Linux AIO support rw flags no 2023-11-08T12:11:50.550 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO support yes 2023-11-08T12:11:50.551 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO support needs -lrt no 2023-11-08T12:11:50.583 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO fsync yes 2023-11-08T12:11:50.614 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX pshared support yes 2023-11-08T12:11:50.658 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_condattr_setclock() yes 2023-11-08T12:11:50.696 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_sigmask() yes 2023-11-08T12:11:50.732 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_getaffinity_np() yes 2023-11-08T12:11:50.742 INFO:teuthology.orchestra.run.smithi049.stdout:Solaris AIO support no 2023-11-08T12:11:50.789 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_fetch_and_add yes 2023-11-08T12:11:50.828 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_synchronize yes 2023-11-08T12:11:50.858 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_val_compare_and_swap yes 2023-11-08T12:11:50.865 INFO:teuthology.orchestra.run.smithi049.stdout:libverbs no 2023-11-08T12:11:50.898 INFO:teuthology.orchestra.run.smithi049.stdout:rdmacm no 2023-11-08T12:11:50.905 INFO:teuthology.orchestra.run.smithi049.stdout:librpma no 2023-11-08T12:11:50.916 INFO:teuthology.orchestra.run.smithi049.stdout:libprotobuf_c no 2023-11-08T12:11:50.946 INFO:teuthology.orchestra.run.smithi049.stdout:asprintf() yes 2023-11-08T12:11:50.978 INFO:teuthology.orchestra.run.smithi049.stdout:vasprintf() yes 2023-11-08T12:11:51.010 INFO:teuthology.orchestra.run.smithi049.stdout:Linux fallocate yes 2023-11-08T12:11:51.042 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX fadvise yes 2023-11-08T12:11:51.074 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX fallocate yes 2023-11-08T12:11:51.105 INFO:teuthology.orchestra.run.smithi049.stdout:sched_setaffinity(3 arg) yes 2023-11-08T12:11:51.105 INFO:teuthology.orchestra.run.smithi049.stdout:sched_setaffinity(2 arg) no 2023-11-08T12:11:51.138 INFO:teuthology.orchestra.run.smithi049.stdout:clock_gettime yes 2023-11-08T12:11:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:CLOCK_MONOTONIC yes 2023-11-08T12:11:51.218 INFO:teuthology.orchestra.run.smithi049.stdout:clockid_t yes 2023-11-08T12:11:51.256 INFO:teuthology.orchestra.run.smithi049.stdout:gettimeofday yes 2023-11-08T12:11:51.289 INFO:teuthology.orchestra.run.smithi049.stdout:fdatasync yes 2023-11-08T12:11:51.321 INFO:teuthology.orchestra.run.smithi049.stdout:pipe() yes 2023-11-08T12:11:51.353 INFO:teuthology.orchestra.run.smithi049.stdout:pipe2() yes 2023-11-08T12:11:51.384 INFO:teuthology.orchestra.run.smithi049.stdout:pread() yes 2023-11-08T12:11:51.470 INFO:teuthology.orchestra.run.smithi049.stdout:sync_file_range yes 2023-11-08T12:11:51.477 INFO:teuthology.orchestra.run.smithi049.stdout:ASharedMemory_create no 2023-11-08T12:11:51.488 INFO:teuthology.orchestra.run.smithi049.stdout:EXT4 move extent yes 2023-11-08T12:11:51.521 INFO:teuthology.orchestra.run.smithi049.stdout:Linux splice(2) yes 2023-11-08T12:11:51.528 INFO:teuthology.orchestra.run.smithi049.stdout:libnuma no 2023-11-08T12:11:51.558 INFO:teuthology.orchestra.run.smithi049.stdout:strsep yes 2023-11-08T12:11:51.589 INFO:teuthology.orchestra.run.smithi049.stdout:strcasestr yes 2023-11-08T12:11:51.602 INFO:teuthology.orchestra.run.smithi049.stdout:strlcat no 2023-11-08T12:11:51.651 INFO:teuthology.orchestra.run.smithi049.stdout:getopt_long_only() yes 2023-11-08T12:11:51.722 INFO:teuthology.orchestra.run.smithi049.stdout:inet_aton yes 2023-11-08T12:11:51.755 INFO:teuthology.orchestra.run.smithi049.stdout:socklen_t yes 2023-11-08T12:11:51.787 INFO:teuthology.orchestra.run.smithi049.stdout:__thread yes 2023-11-08T12:11:51.827 INFO:teuthology.orchestra.run.smithi049.stdout:RUSAGE_THREAD yes 2023-11-08T12:11:51.857 INFO:teuthology.orchestra.run.smithi049.stdout:SCHED_IDLE yes 2023-11-08T12:11:51.893 INFO:teuthology.orchestra.run.smithi049.stdout:TCP_NODELAY yes 2023-11-08T12:11:51.927 INFO:teuthology.orchestra.run.smithi049.stdout:Net engine window_size yes 2023-11-08T12:11:51.963 INFO:teuthology.orchestra.run.smithi049.stdout:TCP_MAXSEG yes 2023-11-08T12:11:51.993 INFO:teuthology.orchestra.run.smithi049.stdout:RLIMIT_MEMLOCK yes 2023-11-08T12:11:52.028 INFO:teuthology.orchestra.run.smithi049.stdout:pwritev/preadv yes 2023-11-08T12:11:52.061 INFO:teuthology.orchestra.run.smithi049.stdout:pwritev2/preadv2 yes 2023-11-08T12:11:52.098 INFO:teuthology.orchestra.run.smithi049.stdout:IPv6 helpers yes 2023-11-08T12:11:52.114 INFO:teuthology.orchestra.run.smithi049.stdout:http engine no 2023-11-08T12:11:52.212 INFO:teuthology.orchestra.run.smithi049.stdout:Rados engine yes 2023-11-08T12:11:52.318 INFO:teuthology.orchestra.run.smithi049.stdout:Rados Block Device engine yes 2023-11-08T12:11:52.436 INFO:teuthology.orchestra.run.smithi049.stdout:rbd_poll yes 2023-11-08T12:11:52.546 INFO:teuthology.orchestra.run.smithi049.stdout:rbd_invalidate_cache yes 2023-11-08T12:11:52.578 INFO:teuthology.orchestra.run.smithi049.stdout:setvbuf yes 2023-11-08T12:11:52.585 INFO:teuthology.orchestra.run.smithi049.stdout:Gluster API engine no 2023-11-08T12:11:52.601 INFO:teuthology.orchestra.run.smithi049.stdout:s390_z196_facilities no 2023-11-08T12:11:52.601 INFO:teuthology.orchestra.run.smithi049.stdout:HDFS engine no 2023-11-08T12:11:52.650 INFO:teuthology.orchestra.run.smithi049.stdout:MTD yes 2023-11-08T12:11:52.659 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem no 2023-11-08T12:11:52.659 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem1_5 no 2023-11-08T12:11:52.666 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem2 no 2023-11-08T12:11:52.667 INFO:teuthology.orchestra.run.smithi049.stdout:PMDK dev-dax engine no 2023-11-08T12:11:52.667 INFO:teuthology.orchestra.run.smithi049.stdout:PMDK libpmem engine no 2023-11-08T12:11:52.675 INFO:teuthology.orchestra.run.smithi049.stdout:DDN's Infinite Memory Engine no 2023-11-08T12:11:52.675 INFO:teuthology.orchestra.run.smithi049.stdout:iscsi engine no 2023-11-08T12:11:52.675 INFO:teuthology.orchestra.run.smithi049.stdout:NBD engine no 2023-11-08T12:11:52.684 INFO:teuthology.orchestra.run.smithi049.stdout:DAOS File System (dfs) Engine no 2023-11-08T12:11:52.693 INFO:teuthology.orchestra.run.smithi049.stdout:NFS engine no 2023-11-08T12:11:52.693 INFO:teuthology.orchestra.run.smithi049.stdout:lex/yacc for arithmetic no 2023-11-08T12:11:52.737 INFO:teuthology.orchestra.run.smithi049.stdout:getmntent yes 2023-11-08T12:11:52.780 INFO:teuthology.orchestra.run.smithi049.stdout:getmntinfo no 2023-11-08T12:11:52.850 INFO:teuthology.orchestra.run.smithi049.stdout:Static Assert yes 2023-11-08T12:11:52.878 INFO:teuthology.orchestra.run.smithi049.stdout:bool yes 2023-11-08T12:11:52.914 INFO:teuthology.orchestra.run.smithi049.stdout:strndup yes 2023-11-08T12:11:52.921 INFO:teuthology.orchestra.run.smithi049.stdout:Valgrind headers no 2023-11-08T12:11:52.962 INFO:teuthology.orchestra.run.smithi049.stdout:Zoned block device support yes 2023-11-08T12:11:52.992 INFO:teuthology.orchestra.run.smithi049.stdout:Zoned block device capacity yes 2023-11-08T12:11:52.999 INFO:teuthology.orchestra.run.smithi049.stdout:libzbc engine no 2023-11-08T12:11:53.048 INFO:teuthology.orchestra.run.smithi049.stdout:NVMe uring command support yes 2023-11-08T12:11:53.063 INFO:teuthology.orchestra.run.smithi049.stdout:xnvme engine no 2023-11-08T12:11:53.070 INFO:teuthology.orchestra.run.smithi049.stdout:isal no 2023-11-08T12:11:53.085 INFO:teuthology.orchestra.run.smithi049.stdout:libblkio engine no 2023-11-08T12:11:53.086 INFO:teuthology.orchestra.run.smithi049.stdout:march_armv8_a_crc_crypto no 2023-11-08T12:11:53.086 INFO:teuthology.orchestra.run.smithi049.stdout:cuda no 2023-11-08T12:11:53.086 INFO:teuthology.orchestra.run.smithi049.stdout:libcufile no 2023-11-08T12:11:53.113 INFO:teuthology.orchestra.run.smithi049.stdout:Build march=native yes 2023-11-08T12:11:53.120 INFO:teuthology.orchestra.run.smithi049.stdout:CUnit no 2023-11-08T12:11:53.150 INFO:teuthology.orchestra.run.smithi049.stdout:__kernel_rwf_t yes 2023-11-08T12:11:53.178 INFO:teuthology.orchestra.run.smithi049.stdout:-Wimplicit-fallthrough=2 yes 2023-11-08T12:11:53.209 INFO:teuthology.orchestra.run.smithi049.stdout:-Wno-stringop-truncation yes 2023-11-08T12:11:53.241 INFO:teuthology.orchestra.run.smithi049.stdout:MADV_HUGEPAGE yes 2023-11-08T12:11:53.273 INFO:teuthology.orchestra.run.smithi049.stdout:gettid yes 2023-11-08T12:11:53.308 INFO:teuthology.orchestra.run.smithi049.stdout:statx(2)/libc yes 2023-11-08T12:11:53.366 INFO:teuthology.orchestra.run.smithi049.stdout:statx(2)/syscall yes 2023-11-08T12:11:53.368 INFO:teuthology.orchestra.run.smithi049.stdout:Windows PDB generation no 2023-11-08T12:11:53.400 INFO:teuthology.orchestra.run.smithi049.stdout:timerfd_create yes 2023-11-08T12:11:53.401 INFO:teuthology.orchestra.run.smithi049.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2023-11-08T12:11:53.402 INFO:teuthology.orchestra.run.smithi049.stdout:Lib-based ioengines dynamic no 2023-11-08T12:11:53.465 INFO:teuthology.orchestra.run.smithi049.stdout:TCMalloc support yes 2023-11-08T12:11:53.467 INFO:teuthology.orchestra.run.smithi049.stdout:seed_buckets 4 2023-11-08T12:11:53.551 INFO:teuthology.orchestra.run.smithi049.stderr:FIO_VERSION = fio-3.36-17-gafdde5 2023-11-08T12:11:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc16.o 2023-11-08T12:11:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32.o 2023-11-08T12:11:53.712 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c-arm64.o 2023-11-08T12:11:53.791 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c-intel.o 2023-11-08T12:11:53.836 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c.o 2023-11-08T12:11:53.876 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc64.o 2023-11-08T12:11:53.917 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc7.o 2023-11-08T12:11:53.951 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crct10dif_common.o 2023-11-08T12:11:53.986 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/fnv.o 2023-11-08T12:11:54.027 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/md5.o 2023-11-08T12:11:54.134 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/murmur3.o 2023-11-08T12:11:54.177 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha1.o 2023-11-08T12:11:54.369 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha256.o 2023-11-08T12:11:54.666 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha3.o 2023-11-08T12:11:54.818 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha512.o 2023-11-08T12:11:54.942 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/test.o 2023-11-08T12:11:55.108 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/xxhash.o 2023-11-08T12:11:55.206 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/axmap.o 2023-11-08T12:11:55.336 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/bloom.o 2023-11-08T12:11:55.421 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/flist_sort.o 2023-11-08T12:11:55.495 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/gauss.o 2023-11-08T12:11:55.595 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/getrusage.o 2023-11-08T12:11:55.631 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/hweight.o 2023-11-08T12:11:55.671 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/ieee754.o 2023-11-08T12:11:55.720 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/lfsr.o 2023-11-08T12:11:55.823 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/memalign.o 2023-11-08T12:11:55.868 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/memcpy.o 2023-11-08T12:11:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/mountcheck.o 2023-11-08T12:11:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/num2str.o 2023-11-08T12:11:56.125 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/output_buffer.o 2023-11-08T12:11:56.174 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/pattern.o 2023-11-08T12:11:56.322 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/prio_tree.o 2023-11-08T12:11:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/rand.o 2023-11-08T12:11:56.638 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/rbtree.o 2023-11-08T12:11:56.764 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/strntol.o 2023-11-08T12:11:56.812 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/zipf.o 2023-11-08T12:11:56.937 INFO:teuthology.orchestra.run.smithi049.stdout: CC gettime.o 2023-11-08T12:11:57.232 INFO:teuthology.orchestra.run.smithi049.stdout: CC ioengines.o 2023-11-08T12:11:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: CC init.o 2023-11-08T12:11:58.303 INFO:teuthology.orchestra.run.smithi049.stdout: CC stat.o 2023-11-08T12:12:00.245 INFO:teuthology.orchestra.run.smithi049.stdout: CC log.o 2023-11-08T12:12:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: CC time.o 2023-11-08T12:12:00.512 INFO:teuthology.orchestra.run.smithi049.stdout: CC filesetup.o 2023-11-08T12:12:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: CC eta.o 2023-11-08T12:12:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: CC verify.o 2023-11-08T12:12:02.275 INFO:teuthology.orchestra.run.smithi049.stdout: CC memory.o 2023-11-08T12:12:02.451 INFO:teuthology.orchestra.run.smithi049.stdout: CC io_u.o 2023-11-08T12:12:03.240 INFO:teuthology.orchestra.run.smithi049.stdout: CC parse.o 2023-11-08T12:12:03.708 INFO:teuthology.orchestra.run.smithi049.stdout: CC fio_sem.o 2023-11-08T12:12:03.818 INFO:teuthology.orchestra.run.smithi049.stdout: CC rwlock.o 2023-11-08T12:12:03.902 INFO:teuthology.orchestra.run.smithi049.stdout: CC pshared.o 2023-11-08T12:12:03.970 INFO:teuthology.orchestra.run.smithi049.stdout: CC options.o 2023-11-08T12:12:04.780 INFO:teuthology.orchestra.run.smithi049.stdout: CC smalloc.o 2023-11-08T12:12:04.968 INFO:teuthology.orchestra.run.smithi049.stdout: CC filehash.o 2023-11-08T12:12:05.126 INFO:teuthology.orchestra.run.smithi049.stdout: CC profile.o 2023-11-08T12:12:05.248 INFO:teuthology.orchestra.run.smithi049.stdout: CC debug.o 2023-11-08T12:12:05.291 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/cpu.o 2023-11-08T12:12:05.432 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/mmap.o 2023-11-08T12:12:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/sync.o 2023-11-08T12:12:05.837 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/null.o 2023-11-08T12:12:05.958 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/net.o 2023-11-08T12:12:06.380 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/ftruncate.o 2023-11-08T12:12:06.484 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/fileoperations.o 2023-11-08T12:12:06.627 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/exec.o 2023-11-08T12:12:06.781 INFO:teuthology.orchestra.run.smithi049.stdout: CC server.o 2023-11-08T12:12:07.545 INFO:teuthology.orchestra.run.smithi049.stdout: CC client.o 2023-11-08T12:12:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: CC iolog.o 2023-11-08T12:12:08.566 INFO:teuthology.orchestra.run.smithi049.stdout: CC backend.o 2023-11-08T12:12:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: CC libfio.o 2023-11-08T12:12:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: CC flow.o 2023-11-08T12:12:09.749 INFO:teuthology.orchestra.run.smithi049.stdout: CC cconv.o 2023-11-08T12:12:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: CC gettime-thread.o 2023-11-08T12:12:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: CC helpers.o 2023-11-08T12:12:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: CC json.o 2023-11-08T12:12:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: CC idletime.o 2023-11-08T12:12:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: CC td_error.o 2023-11-08T12:12:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: CC profiles/tiobench.o 2023-11-08T12:12:10.910 INFO:teuthology.orchestra.run.smithi049.stdout: CC profiles/act.o 2023-11-08T12:12:11.072 INFO:teuthology.orchestra.run.smithi049.stdout: CC io_u_queue.o 2023-11-08T12:12:11.123 INFO:teuthology.orchestra.run.smithi049.stdout: CC filelock.o 2023-11-08T12:12:11.253 INFO:teuthology.orchestra.run.smithi049.stdout: CC workqueue.o 2023-11-08T12:12:11.436 INFO:teuthology.orchestra.run.smithi049.stdout: CC rate-submit.o 2023-11-08T12:12:11.587 INFO:teuthology.orchestra.run.smithi049.stdout: CC optgroup.o 2023-11-08T12:12:11.633 INFO:teuthology.orchestra.run.smithi049.stdout: CC helper_thread.o 2023-11-08T12:12:11.821 INFO:teuthology.orchestra.run.smithi049.stdout: CC steadystate.o 2023-11-08T12:12:12.062 INFO:teuthology.orchestra.run.smithi049.stdout: CC zone-dist.o 2023-11-08T12:12:12.203 INFO:teuthology.orchestra.run.smithi049.stdout: CC zbd.o 2023-11-08T12:12:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: CC dedupe.o 2023-11-08T12:12:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: CC fdp.o 2023-11-08T12:12:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/posixaio.o 2023-11-08T12:12:13.343 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/falloc.o 2023-11-08T12:12:13.456 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/e4defrag.o 2023-11-08T12:12:13.589 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/splice.o 2023-11-08T12:12:13.784 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/asprintf.o 2023-11-08T12:12:13.821 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strlcat.o 2023-11-08T12:12:13.865 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/mtd.o 2023-11-08T12:12:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/libmtd.o 2023-11-08T12:12:14.425 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/libmtd_legacy.o 2023-11-08T12:12:14.561 INFO:teuthology.orchestra.run.smithi049.stdout: CC diskutil.o 2023-11-08T12:12:14.776 INFO:teuthology.orchestra.run.smithi049.stdout: CC fifo.o 2023-11-08T12:12:14.833 INFO:teuthology.orchestra.run.smithi049.stdout: CC blktrace.o 2023-11-08T12:12:15.171 INFO:teuthology.orchestra.run.smithi049.stdout: CC cgroup.o 2023-11-08T12:12:15.329 INFO:teuthology.orchestra.run.smithi049.stdout: CC trim.o 2023-11-08T12:12:15.445 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/sg.o 2023-11-08T12:12:15.837 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/linux-dev-lookup.o 2023-11-08T12:12:15.895 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/io_uring.o 2023-11-08T12:12:16.287 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/nvme.o 2023-11-08T12:12:16.589 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/linux-blkzoned.o 2023-11-08T12:12:16.749 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/cmdprio.o 2023-11-08T12:12:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/rados.o 2023-11-08T12:12:17.178 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/rbd.o 2023-11-08T12:12:17.368 INFO:teuthology.orchestra.run.smithi049.stdout: CC fio.o 2023-11-08T12:12:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: LINK fio 2023-11-08T12:12:17.658 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/log.o 2023-11-08T12:12:17.700 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/genzipf.o 2023-11-08T12:12:17.841 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strcasestr.o 2023-11-08T12:12:17.866 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strndup.o 2023-11-08T12:12:17.890 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-genzipf 2023-11-08T12:12:17.981 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/btrace2fio.o 2023-11-08T12:12:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-btrace2fio 2023-11-08T12:12:18.519 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/verify-state.o 2023-11-08T12:12:18.623 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/debug.o 2023-11-08T12:12:18.659 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-verify-state 2023-11-08T12:12:18.749 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/stest.o 2023-11-08T12:12:18.819 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/arch.o 2023-11-08T12:12:18.850 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/stest 2023-11-08T12:12:18.942 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/ieee754.o 2023-11-08T12:12:18.983 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/ieee754 2023-11-08T12:12:19.073 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/axmap.o 2023-11-08T12:12:19.177 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/axmap 2023-11-08T12:12:19.267 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/lfsr-test.o 2023-11-08T12:12:19.366 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/lfsr-test 2023-11-08T12:12:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/gen-rand.o 2023-11-08T12:12:19.560 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/gen-rand 2023-11-08T12:12:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/memlock.o 2023-11-08T12:12:19.719 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/memlock 2023-11-08T12:12:19.807 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/read-to-pipe-async.o 2023-11-08T12:12:20.004 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/read-to-pipe-async 2023-11-08T12:12:20.093 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/io_uring.o 2023-11-08T12:12:20.650 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/io_uring 2023-11-08T12:12:20.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-11-08T12:12:20.774 DEBUG:teuthology.orchestra.run.smithi049:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-08T12:12:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - Settings.cluster: 2023-11-08T12:12:21.024 INFO:teuthology.orchestra.run.smithi049.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-11-08T12:12:21.025 INFO:teuthology.orchestra.run.smithi049.stderr: 'clients': ['ubuntu@smithi049.front.sepia.ceph.com'], 2023-11-08T12:12:21.025 INFO:teuthology.orchestra.run.smithi049.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-11-08T12:12:21.025 INFO:teuthology.orchestra.run.smithi049.stderr: 'head': 'ubuntu@smithi049.front.sepia.ceph.com', 2023-11-08T12:12:21.026 INFO:teuthology.orchestra.run.smithi049.stderr: 'is_teuthology': True, 2023-11-08T12:12:21.026 INFO:teuthology.orchestra.run.smithi049.stderr: 'iterations': 1, 2023-11-08T12:12:21.026 INFO:teuthology.orchestra.run.smithi049.stderr: 'mons': ['ubuntu@smithi049.front.sepia.ceph.com'], 2023-11-08T12:12:21.026 INFO:teuthology.orchestra.run.smithi049.stderr: 'osds': ['ubuntu@smithi049.front.sepia.ceph.com'], 2023-11-08T12:12:21.027 INFO:teuthology.orchestra.run.smithi049.stderr: 'osds_per_node': 3, 2023-11-08T12:12:21.027 INFO:teuthology.orchestra.run.smithi049.stderr: 'pid_dir': '/var/run/ceph', 2023-11-08T12:12:21.027 INFO:teuthology.orchestra.run.smithi049.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2023-11-08T12:12:21.028 INFO:teuthology.orchestra.run.smithi049.stderr: 'rebuild_every_test': False, 2023-11-08T12:12:21.028 INFO:teuthology.orchestra.run.smithi049.stderr: 'rgws': {'ubuntu@smithi049.front.sepia.ceph.com': None}, 2023-11-08T12:12:21.028 INFO:teuthology.orchestra.run.smithi049.stderr: 'tmp_dir': '/tmp/cbt', 2023-11-08T12:12:21.029 INFO:teuthology.orchestra.run.smithi049.stderr: 'use_existing': True, 2023-11-08T12:12:21.029 INFO:teuthology.orchestra.run.smithi049.stderr: 'user': 'ubuntu'} 2023-11-08T12:12:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi 2023-11-08T12:12:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2023-11-08T12:12:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-11-08T12:12:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2023-11-08T12:12:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-08T12:12:21.035 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.035 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.036 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2023-11-08T12:12:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:12:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-08T12:12:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:12:12:21 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-08T12:13:21.075 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.076 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.076 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-08T12:13:21.113 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.114 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.114 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-08T12:13:21.155 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.155 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.155 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2023-11-08T12:13:21.176 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.176 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/$(basename ${i}).smithi049.front.sepia.ceph.com ; done 2023-11-08T12:13:21.180 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.180 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.180 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2023-11-08T12:13:21.183 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.183 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-08T12:13:21.184 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.185 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.185 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2023-11-08T12:13:21.188 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.188 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-08T12:13:21.188 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-08T12:13:21.189 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.189 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.189 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2023-11-08T12:13:21.494 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.494 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:21.494 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2023-11-08T12:13:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:22 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:22 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:22 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2023-11-08T12:13:23.219 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:23 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:23.220 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:23 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:23.220 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2023-11-08T12:13:24.225 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:24 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:24.225 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:24 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:24.226 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2023-11-08T12:13:25.226 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:25 - INFO - cbt - Checking Health after pool creation. 2023-11-08T12:13:30.231 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:30.232 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:30 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:30.232 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-08T12:13:30.564 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:30 - INFO - cbt - Final Pool Health Check. 2023-11-08T12:13:35.570 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:35.571 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:35.571 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-08T12:13:35.892 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:35.893 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:35.893 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2023-11-08T12:13:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi049.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2023-11-08T12:13:37.640 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.640 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.640 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi049.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2023-11-08T12:13:37.737 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.737 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.738 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-08T12:13:37.756 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.756 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.757 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-08T12:13:37.790 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - INFO - cbt - Attempting to populating fio files... 2023-11-08T12:13:37.791 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.791 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.791 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-11-08T12:13:37.792 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.792 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:13:37.792 INFO:teuthology.orchestra.run.smithi049.stderr:12:13:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-11-08T12:16:12.394 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.395 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433. 2023-11-08T12:16:12.395 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - INFO - cbt - Setting OSD Read Ahead to: 4096 2023-11-08T12:16:12.395 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.396 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.396 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2023-11-08T12:16:12.397 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2023-11-08T12:16:12.397 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.397 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.397 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2023-11-08T12:16:12.415 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.415 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:12.416 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-08T12:16:13.078 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:13 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:13.078 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-08T12:16:13.865 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:13 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:13.866 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:13 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:13.866 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2023-11-08T12:16:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:14 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:14 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2023-11-08T12:16:39.921 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:39 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:39 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:39.926 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2023-11-08T12:16:40.264 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.265 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.265 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/collectl 2023-11-08T12:16:40.267 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.267 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-08T12:16:40.269 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.269 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.269 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf 2023-11-08T12:16:40.276 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.276 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-08T12:16:40.276 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-08T12:16:40.276 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - INFO - cbt - Running rbd fio randread test. 2023-11-08T12:16:40.277 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2023-11-08T12:16:40.277 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.277 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.277 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 2023-11-08T12:16:40.277 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2023-11-08T12:16:40.278 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.278 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:40.278 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 2023-11-08T12:16:58.181 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.181 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.181 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-08T12:16:58.200 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.201 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.201 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-08T12:16:58.232 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.232 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2023-11-08T12:16:58.233 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.234 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2023-11-08T12:16:58.235 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.235 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.236 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2023-11-08T12:16:58.550 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.570 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.570 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2023-11-08T12:16:58.588 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.589 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/$(basename ${i}).smithi049.front.sepia.ceph.com ; done 2023-11-08T12:16:58.621 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - INFO - cbt - Convert results to json format. 2023-11-08T12:16:58.621 INFO:teuthology.orchestra.run.smithi049.stderr:12:16:58 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.622 INFO:teuthology.orchestra.run.smithi049.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2023-11-08T12:16:58.622 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049.front.sepia.ceph.com: smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.622 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-08T12:16:58.622 INFO:teuthology.orchestra.run.smithi049.stdout:['smithi049.front.sepia.ceph.com'] 2023-11-08T12:16:58.644 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-11-08T12:16:58.654 DEBUG:teuthology.orchestra.run.smithi049:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-11-08T12:16:58.824 DEBUG:teuthology.orchestra.run.smithi049:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2023-11-08T12:16:58.889 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-11-08T12:16:58.889 INFO:tasks.cbt_performance:reading cbt preferences 2023-11-08T12:16:58.890 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-11-08T12:16:58.925 INFO:teuthology.orchestra.run.smithi049.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.0.smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.925 INFO:teuthology.orchestra.run.smithi049.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.1.smithi049.front.sepia.ceph.com 2023-11-08T12:16:58.926 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:16:58.927 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.0.smithi049.front.sepia.ceph.com of=/dev/stdout 2023-11-08T12:16:58.983 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-08T12:16:58.983 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 -name 'perf_stat.*' 2023-11-08T12:16:59.038 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-08T12:16:59.038 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-08T12:16:59.039 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-08T12:16:59.039 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.1.smithi049.front.sepia.ceph.com of=/dev/stdout 2023-11-08T12:16:59.094 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-08T12:16:59.094 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 -name 'perf_stat.*' 2023-11-08T12:16:59.149 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-08T12:16:59.150 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-08T12:16:59.160 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-08T12:16:59.170 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-08T12:16:59.170 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-08T12:16:59.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-08T12:16:59.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:16:59.295+0000 7efe86b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:16:59.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-08T12:16:59.302+0000 7efe86b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:16:59.505 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-08T12:16:59.506 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-08T12:16:59.506 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-08T12:16:59.506 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:17:03.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-08T12:17:05.608 INFO:tasks.ceph.osd.0:Stopped 2023-11-08T12:17:05.609 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-08T12:17:05.609 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:17:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-08T12:17:08.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-11-08T12:17:11.710 INFO:tasks.ceph.osd.1:Stopped 2023-11-08T12:17:11.712 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-08T12:17:11.712 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:17:13.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-08T12:17:13.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-11-08T12:17:17.814 INFO:tasks.ceph.osd.2:Stopped 2023-11-08T12:17:17.815 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-08T12:17:17.817 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-08T12:17:17.817 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:17:17.849 INFO:tasks.ceph.mgr.x:Stopped 2023-11-08T12:17:17.850 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-08T12:17:17.850 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-08T12:17:17.850 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:17:17.903 INFO:tasks.ceph.mon.a:Stopped 2023-11-08T12:17:17.904 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-08T12:17:17.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-08T12:17:18.022 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:17:18.023 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-08T12:17:18.147 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:17:18.148 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-08T12:17:18.367 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:17:18.368 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-08T12:17:18.433 INFO:tasks.ceph:Archiving mon data... 2023-11-08T12:17:18.434 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916/data/mon.a.tgz 2023-11-08T12:17:18.434 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-11-08T12:17:18.457 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.wrJWGtNSJa 2023-11-08T12:17:18.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wrJWGtNSJa 2023-11-08T12:17:18.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.wrJWGtNSJa 2023-11-08T12:17:18.711 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.wrJWGtNSJa is 198KB 2023-11-08T12:17:18.768 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.wrJWGtNSJa 2023-11-08T12:17:18.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-08T12:17:18.784 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-08T12:17:18.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T12:17:18.855 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-08T12:17:18.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-08T12:17:19.049 INFO:teuthology.orchestra.run.smithi049.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-08T12:17:19.053 INFO:tasks.ceph:Archiving crash dumps... 2023-11-08T12:17:19.055 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916/remote/smithi049/crash 2023-11-08T12:17:19.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-08T12:17:19.087 INFO:tasks.ceph:Compressing logs... 2023-11-08T12:17:19.087 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-08T12:17:19.180 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-08T12:17:19.180 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82250.log 2023-11-08T12:17:19.181 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82043.log 2023-11-08T12:17:19.181 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86206.log 2023-11-08T12:17:19.181 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82123.log 2023-11-08T12:17:19.181 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.86206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85740.log 2023-11-08T12:17:19.181 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86206.log.gz 2023-11-08T12:17:19.182 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81983.log 2023-11-08T12:17:19.182 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85786.log 2023-11-08T12:17:19.182 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85810.log 2023-11-08T12:17:19.182 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80927.log 2023-11-08T12:17:19.182 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85810.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79556.log 2023-11-08T12:17:19.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.80927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82125.log 2023-11-08T12:17:19.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.tmp-client.admin.79556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82211.log 2023-11-08T12:17:19.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82421.log 2023-11-08T12:17:19.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85599.log 2023-11-08T12:17:19.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85859.log 2023-11-08T12:17:19.185 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85858.log 2023-11-08T12:17:19.185 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85663.log 2023-11-08T12:17:19.185 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85858.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-08T12:17:19.185 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82043.log.gz 2023-11-08T12:17:19.186 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81983.log.gz 2023-11-08T12:17:19.186 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82123.log.gz 2023-11-08T12:17:19.186 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81125.log 2023-11-08T12:17:19.186 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85786.log.gz 2023-11-08T12:17:19.186 INFO:teuthology.orchestra.run.smithi049.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.85810.log.gz 2023-11-08T12:17:19.187 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82251.log 2023-11-08T12:17:19.187 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82125.log.gz 2023-11-08T12:17:19.187 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82211.log.gz 2023-11-08T12:17:19.187 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82250.log.gz 2023-11-08T12:17:19.188 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81125.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-08T12:17:19.188 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82421.log.gz 2023-11-08T12:17:19.188 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81940.log 2023-11-08T12:17:19.188 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81856.log 2023-11-08T12:17:19.188 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.81940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85715.log 2023-11-08T12:17:19.190 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81363.log 2023-11-08T12:17:19.190 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81125.log.gz 2023-11-08T12:17:19.190 INFO:teuthology.orchestra.run.smithi049.stderr: 85.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-08T12:17:19.190 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85715.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81940.log.gz 2023-11-08T12:17:19.191 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81393.log 2023-11-08T12:17:19.191 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.82251.log.gz --verbose -- 2023-11-08T12:17:19.191 INFO:teuthology.orchestra.run.smithi049.stderr: /var/log/ceph/ceph-client.admin.81015.log 2023-11-08T12:17:19.191 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81363.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81856.log.gz 2023-11-08T12:17:19.191 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81800.log 2023-11-08T12:17:19.192 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82124.log 2023-11-08T12:17:19.192 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81800.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-08T12:17:19.192 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86207.log 2023-11-08T12:17:19.192 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81015.log.gz 2023-11-08T12:17:19.193 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82337.log 2023-11-08T12:17:19.193 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.86207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81165.log 2023-11-08T12:17:19.193 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86207.log.gz 2023-11-08T12:17:19.193 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81800.log.gz 2023-11-08T12:17:19.193 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82337.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-08T12:17:19.194 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81165.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82124.log.gz 2023-11-08T12:17:19.194 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85690.log 2023-11-08T12:17:19.194 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82337.log.gz 2023-11-08T12:17:19.194 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.86159.log 2023-11-08T12:17:19.195 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82379.log 2023-11-08T12:17:19.195 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.86159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85763.log 2023-11-08T12:17:19.195 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86159.log.gz 2023-11-08T12:17:19.195 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82379.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-08T12:17:19.195 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82379.log.gz 2023-11-08T12:17:19.196 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81898.log 2023-11-08T12:17:19.196 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85763.log: /var/log/ceph/ceph.log: gzip 50.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.85763.log.gz 2023-11-08T12:17:19.196 INFO:teuthology.orchestra.run.smithi049.stderr: -- /var/log/ceph/ceph-osd.2.log 2023-11-08T12:17:19.196 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81205.log 2023-11-08T12:17:19.196 INFO:teuthology.orchestra.run.smithi049.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-08T12:17:19.196 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86493.log 2023-11-08T12:17:19.197 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81898.log.gz 2023-11-08T12:17:19.197 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.81205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85827.log 2023-11-08T12:17:19.197 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.86493.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86493.log.gz 2023-11-08T12:17:19.197 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85827.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85859.log.gz 2023-11-08T12:17:19.198 INFO:teuthology.orchestra.run.smithi049.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.85827.log.gz 2023-11-08T12:17:19.199 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85858.log.gz 2023-11-08T12:17:19.199 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81165.log.gz 2023-11-08T12:17:19.199 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81205.log.gz 2023-11-08T12:17:19.199 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81363.log.gz 2023-11-08T12:17:19.199 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81393.log.gz 2023-11-08T12:17:19.271 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85599.log.gz 2023-11-08T12:17:19.271 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85663.log.gz 2023-11-08T12:17:19.271 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85690.log.gz 2023-11-08T12:17:19.271 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85715.log.gz 2023-11-08T12:17:19.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85740.log.gz 2023-11-08T12:17:19.344 INFO:teuthology.orchestra.run.smithi049.stderr: 66.9% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79556.log.gz 2023-11-08T12:17:19.353 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80927.log.gz 2023-11-08T12:17:19.354 INFO:teuthology.orchestra.run.smithi049.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-08T12:17:19.380 INFO:teuthology.orchestra.run.smithi049.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-08T12:17:36.703 INFO:teuthology.orchestra.run.smithi049.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-08T12:17:37.200 INFO:teuthology.orchestra.run.smithi049.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-08T12:17:37.546 INFO:teuthology.orchestra.run.smithi049.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-08T12:17:37.548 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-08T12:17:37.548 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m18.406s 2023-11-08T12:17:37.549 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m10.849s 2023-11-08T12:17:37.549 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.575s 2023-11-08T12:17:37.550 INFO:tasks.ceph:Archiving logs... 2023-11-08T12:17:37.550 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916/remote/smithi049/log 2023-11-08T12:17:37.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-08T12:17:39.101 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-08T12:17:39.111 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-08T12:17:39.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-08T12:17:39.196 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-08T12:17:39.205 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-08T12:17:39.206 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T12:17:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2023-11-08T12:17:39.283 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T12:17:39.284 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-11-08T12:17:39.284 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +885us[ +885us] +/- 70ms 2023-11-08T12:17:39.284 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -291us[ -288us] +/- 75ms 2023-11-08T12:17:39.284 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 -396us[ -396us] +/- 68ms 2023-11-08T12:17:39.284 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:17:39.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-08T12:17:39.304 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-08T12:17:39.305 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-08T12:17:39.325 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-08T12:17:39.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T12:17:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:17:39.554 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-08T12:17:39.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-08T12:17:39.572 INFO:teuthology.task.internal:Duration was 918.228088 seconds 2023-11-08T12:17:39.573 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-08T12:17:39.585 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-08T12:17:39.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-08T12:17:39.706 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T12:17:40.146 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-08T12:17:40.147 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2023-11-08T12:17:40.147 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-08T12:17:40.168 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-08T12:17:40.168 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T12:17:40.241 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-08T12:17:40.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-08T12:17:40.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-08T12:17:40.409 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-08T12:17:40.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-08T12:17:40.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-08T12:17:40.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-08T12:17:40.526 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2023-11-08T12:17:40.560 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-08T12:17:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:17:40.575 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-08T12:17:40.586 INFO:teuthology.task.internal:Transferring archived files... 2023-11-08T12:17:40.587 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451916/remote/smithi049 2023-11-08T12:17:40.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-08T12:17:40.689 INFO:teuthology.task.internal:Removing archive directory... 2023-11-08T12:17:40.689 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-08T12:17:40.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-08T12:17:40.737 INFO:teuthology.task.internal:Not uploading archives. 2023-11-08T12:17:40.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-08T12:17:40.747 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-08T12:17:40.748 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-08T12:17:40.779 INFO:teuthology.orchestra.run.smithi049.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 8 12:17 /home/ubuntu/cephtest 2023-11-08T12:17:40.780 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-08T12:17:40.791 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-08T12:17:40.815 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-matanb-crimson-do_osd_ops_execute-v3&var-job_id_selected=7451916 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} duration: 918.2280881404877 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-11-08T12:17:40.816 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T12:17:40.888 INFO:teuthology.run:pass