2023-10-31T11:50:44.918 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-31T11:50:44.919 DEBUG:teuthology.run:Teuthology command: teuthology --name matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957 --owner scheduled_matan@teuthology --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} -- /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957/orig.config.yaml 2023-10-31T11:50:44.962 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-31T11:50:45.041 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957 branch: wip-wip-matanb-crimson-do_osd_ops_execute-v3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} email: null first_in_suite: false job_id: '7441957' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-wip-matanb-crimson-do_osd_ops_execute-v3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: e83bf4821bfb604694b03a67d7390bfb38655506 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: e83bf4821bfb604694b03a67d7390bfb38655506 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 21548fe806cf259deac1421530d5ce720be17997 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1530 sha1: e83bf4821bfb604694b03a67d7390bfb38655506 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 21548fe806cf259deac1421530d5ce720be17997 targets: smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIWtOzSIuJbVsTmU7duj9X/WJldXEx3LDhPSRdW79v7wTLOUEUP7milEjJtJazDCQFmwdeXNOi1OaQUkEsW4WxE= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-31_11:40:33 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-31T11:50:45.041 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa; will attempt to use it 2023-10-31T11:50:45.042 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa/tasks 2023-10-31T11:50:45.043 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-31T11:50:45.044 INFO:teuthology.task.internal:Checking packages... 2023-10-31T11:50:45.067 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'e83bf4821bfb604694b03a67d7390bfb38655506' 2023-10-31T11:50:45.068 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-31T11:50:45.068 INFO:teuthology.packaging:ref: None 2023-10-31T11:50:45.069 INFO:teuthology.packaging:tag: None 2023-10-31T11:50:45.069 INFO:teuthology.packaging:branch: wip-wip-matanb-crimson-do_osd_ops_execute-v3 2023-10-31T11:50:45.069 INFO:teuthology.packaging:sha1: e83bf4821bfb604694b03a67d7390bfb38655506 2023-10-31T11:50:45.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-wip-matanb-crimson-do_osd_ops_execute-v3 2023-10-31T11:50:45.223 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6792.ge83bf482 2023-10-31T11:50:45.223 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-31T11:50:45.230 INFO:teuthology.task.internal:no buildpackages task found 2023-10-31T11:50:45.231 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-31T11:50:45.245 INFO:teuthology.task.internal:Saving configuration 2023-10-31T11:50:45.262 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-31T11:50:45.267 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-31T11:50:45.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-31 11:45:31.780638', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIWtOzSIuJbVsTmU7duj9X/WJldXEx3LDhPSRdW79v7wTLOUEUP7milEjJtJazDCQFmwdeXNOi1OaQUkEsW4WxE='} 2023-10-31T11:50:45.294 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-31T11:50:45.300 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-31T11:50:45.300 INFO:teuthology.run_tasks:Running task console_log... 2023-10-31T11:50:45.338 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa2add83940>, signals=[15]) 2023-10-31T11:50:45.339 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-31T11:50:45.345 INFO:teuthology.task.internal:Opening connections... 2023-10-31T11:50:45.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T11:50:45.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T11:50:45.430 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-31T11:50:45.437 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2023-10-31T11:50:45.499 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2023-10-31T11:50:45.499 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2023-10-31T11:50:45.556 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2023-10-31T11:50:45.556 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2023-10-31T11:50:45.556 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2023-10-31T11:50:45.557 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2023-10-31T11:50:45.557 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2023-10-31T11:50:45.557 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2023-10-31T11:50:45.557 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-31T11:50:45.558 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2023-10-31T11:50:45.558 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2023-10-31T11:50:45.558 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-31T11:50:45.558 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2023-10-31T11:50:45.559 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-31T11:50:45.559 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-31T11:50:45.559 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-31T11:50:45.560 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2023-10-31T11:50:45.582 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-31T11:50:45.591 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-31T11:50:45.597 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-31T11:50:45.598 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2023-10-31T11:50:45.614 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-31T11:50:45.620 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-31T11:50:45.620 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2023-10-31T11:50:45.913 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-31T11:50:45.947 INFO:teuthology.run_tasks:Running task kernel... 2023-10-31T11:50:45.960 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-31T11:50:45.961 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-31T11:50:45.962 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-31T11:50:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T11:50:45.979 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2023-10-31T11:50:46.036 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-361.el9.x86_64 2023-10-31T11:50:46.037 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.14.0-361.el9.x86_64 2023-10-31T11:50:46.037 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2023-10-31T11:50:48.217 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-31T11:50:49.584 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2023-10-31T11:50:50.458 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-10-31T11:50:53.451 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 18 MB 00:01 2023-10-31T11:51:00.643 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 5.8 MB 00:03 2023-10-31T11:51:02.052 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2023-10-31T11:51:03.914 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-10-31T11:51:08.412 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2023-10-31T11:51:10.468 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-31T11:51:10.468 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-31T11:51:10.563 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T11:51:10.568 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T11:51:10.568 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2023-10-31T11:51:10.568 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T11:51:10.569 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T11:51:10.569 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-31T11:51:10.569 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T11:51:10.569 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-31T11:51:10.570 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-31T11:51:10.570 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-31T11:51:10.570 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T11:51:10.570 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T11:51:10.571 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T11:51:10.571 INFO:teuthology.orchestra.run.smithi161.stdout:Install 4 Packages 2023-10-31T11:51:10.571 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T11:51:10.572 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 95 M 2023-10-31T11:51:10.572 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 120 M 2023-10-31T11:51:10.572 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T11:51:14.196 INFO:teuthology.orchestra.run.smithi161.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 1.5 MB/s | 5.3 MB 00:03 2023-10-31T11:51:22.987 INFO:teuthology.orchestra.run.smithi161.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 1.6 MB/s | 20 MB 00:12 2023-10-31T11:51:25.187 INFO:teuthology.orchestra.run.smithi161.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 2.6 MB/s | 38 MB 00:14 2023-10-31T11:51:29.078 INFO:teuthology.orchestra.run.smithi161.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 2.2 MB/s | 32 MB 00:14 2023-10-31T11:51:29.079 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T11:51:29.079 INFO:teuthology.orchestra.run.smithi161.stdout:Total 5.1 MB/s | 95 MB 00:18 2023-10-31T11:51:29.927 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T11:51:30.289 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T11:51:30.289 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T11:51:31.569 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T11:51:31.570 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T11:51:34.189 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T11:51:34.883 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-31T11:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T11:51:37.858 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T11:51:38.040 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T11:51:42.674 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T11:51:42.797 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-31T11:51:47.458 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T11:52:16.488 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T11:52:16.531 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-31T11:52:16.970 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-31T11:52:16.970 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-31T11:52:16.971 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T11:52:16.971 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T11:52:17.448 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T11:52:17.448 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T11:52:17.448 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T11:52:17.449 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-31T11:52:17.449 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-31T11:52:17.449 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-31T11:52:17.449 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-31T11:52:17.449 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T11:52:17.450 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T11:52:17.924 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2023-10-31T11:52:18.083 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-31T11:52:18.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-31T11:52:18.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-31T11:52:18.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-31T11:52:18.085 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-31T11:52:18.085 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-31T11:52:18.085 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2023-10-31T11:52:18.085 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2023-10-31T11:52:18.203 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-31T11:52:18.203 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-31T11:52:18.203 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2023-10-31T11:52:18.247 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2023-10-31T11:52:18.247 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2023-10-31T11:52:18.247 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.06 2023-10-31T11:52:18.247 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 68.el9 2023-10-31T11:52:18.248 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2023-10-31T11:52:18.248 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-31T11:52:18.248 INFO:teuthology.orchestra.run.smithi161.stdout:Group : Unspecified 2023-10-31T11:52:18.248 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 8264254 2023-10-31T11:52:18.248 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2023-10-31T11:52:18.249 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-31T11:52:18.249 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-31T11:52:18.249 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-31T11:52:18.249 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-31T11:52:18.249 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : builder@centos.org 2023-10-31T11:52:18.250 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2023-10-31T11:52:18.250 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-31T11:52:18.250 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2023-10-31T11:52:18.250 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2023-10-31T11:52:18.250 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T11:52:18.251 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-31T11:52:18.251 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-31T11:52:18.251 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-31T11:52:18.251 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2023-10-31T11:52:18.251 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T11:52:18.252 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2023-10-31T11:52:18.252 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-31T11:52:18.253 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 5.14.0-378.el9.x86_64 2023-10-31T11:52:18.253 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-31T11:52:18.940 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2023-10-31T11:52:20.377 INFO:teuthology.orchestra.run.smithi161.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-31T11:52:20.406 INFO:teuthology.orchestra.run.smithi161.stderr:done 2023-10-31T11:52:20.408 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2023-10-31T11:52:20.464 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.2LJszJFY2d 2023-10-31T11:52:20.464 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2LJszJFY2d 2023-10-31T11:52:20.533 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.2LJszJFY2d 2023-10-31T11:52:20.665 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.2LJszJFY2d is 6KB 2023-10-31T11:52:20.715 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.2LJszJFY2d 2023-10-31T11:52:20.730 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2023-10-31T11:52:20.794 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-31T11:52:20.795 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-31T11:52:20.795 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-31T11:52:20.795 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-31T11:52:20.796 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-31T11:52:20.955 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2023-10-31T11:52:50.986 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-31T11:52:50.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T11:52:50.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T11:53:09.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2023-10-31T11:53:18.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T11:53:18.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T11:53:21.562 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2023-10-31T11:53:33.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T11:53:33.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T11:54:33.636 DEBUG:teuthology.orchestra.remote:timed out 2023-10-31T11:54:48.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T11:54:48.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T11:54:49.068 DEBUG:teuthology.orchestra.run.smithi161:> true 2023-10-31T11:54:49.416 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T11:54:49.417 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-31T11:54:49.417 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2023-10-31T11:54:49.433 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-378.el9.x86_64 2023-10-31T11:54:49.433 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-31T11:54:49.433 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-31T11:54:49.433 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-31T11:54:50.435 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-31T11:54:50.435 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-31T11:54:50.528 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2023-10-31T11:54:50.554 DEBUG:teuthology.parallel:result is None 2023-10-31T11:54:50.555 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-31T11:54:50.561 INFO:teuthology.task.internal:Creating test directory... 2023-10-31T11:54:50.562 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-31T11:54:50.579 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-31T11:54:50.585 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-31T11:54:50.594 INFO:teuthology.task.internal:Creating archive directory... 2023-10-31T11:54:50.595 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-31T11:54:50.648 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-31T11:54:50.655 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-31T11:54:50.655 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-31T11:54:50.720 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-31T11:54:50.733 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-31T11:54:50.735 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-31T11:54:50.743 INFO:teuthology.task.internal:Configuring sudo... 2023-10-31T11:54:50.743 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-31T11:54:50.803 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-31T11:54:50.812 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-31T11:54:50.813 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-31T11:54:50.859 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-31T11:54:50.945 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-31T11:54:50.993 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-31T11:54:51.074 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-31T11:54:51.117 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T11:54:51.117 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-31T11:54:51.183 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2023-10-31T11:54:51.271 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-31T11:54:51.528 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-31T11:54:51.536 INFO:teuthology.task.internal:Starting timer... 2023-10-31T11:54:51.537 INFO:teuthology.run_tasks:Running task pcp... 2023-10-31T11:54:51.548 INFO:teuthology.run_tasks:Running task selinux... 2023-10-31T11:54:51.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2023-10-31T11:54:51.608 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2023-10-31T11:54:51.610 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-31T11:54:51.611 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2023-10-31T11:54:51.636 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2023-10-31T11:54:51.636 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2023-10-31T11:54:51.637 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-31T11:54:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T11:54:51.704 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-31T11:54:51.704 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2023-10-31T11:54:51.778 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-31T11:54:51.790 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-31T11:54:51.791 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-31T11:54:51.806 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-31T11:54:51.807 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-31T12:00:14.833 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2023-10-31T12:00:14.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T12:00:14.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:00:14.921 DEBUG:teuthology.orchestra.run.smithi161:> true 2023-10-31T12:00:14.989 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2023-10-31T12:00:14.990 INFO:teuthology.run_tasks:Running task clock... 2023-10-31T12:00:15.011 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-31T12:00:15.012 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-31T12:00:15.013 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-31T12:00:15.060 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-31T12:00:15.077 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-31T12:00:15.106 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2023-10-31T12:00:15.119 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2023-10-31T12:00:15.135 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-31T12:00:15.151 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-31T12:00:15.194 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2023-10-31T12:00:15.197 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-31T12:00:15.197 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2023-10-31T12:00:15.198 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:00:15.198 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:00:15.198 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:00:15.198 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:00:15.199 INFO:teuthology.run_tasks:Running task install... 2023-10-31T12:00:15.208 DEBUG:teuthology.task.install:project ceph 2023-10-31T12:00:15.208 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'}} 2023-10-31T12:00:15.208 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'} 2023-10-31T12:00:15.209 INFO:teuthology.task.install:Using flavor: crimson 2023-10-31T12:00:15.214 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-31T12:00:15.215 INFO:teuthology.task.install:extra packages: [] 2023-10-31T12:00:15.215 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506', 'tag': None, 'wait_for_package': False} 2023-10-31T12:00:15.216 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=e83bf4821bfb604694b03a67d7390bfb38655506 2023-10-31T12:00:15.365 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/ 2023-10-31T12:00:15.366 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6792.ge83bf482 2023-10-31T12:00:15.475 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-31T12:00:15.476 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:00:15.476 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-31T12:00:15.503 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-31T12:00:15.504 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e83bf4821bfb604694b03a67d7390bfb38655506/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-31T12:00:15.572 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-31T12:00:15.652 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-31T12:00:15.718 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2023-10-31T12:00:15.719 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2023-10-31T12:00:15.966 INFO:teuthology.orchestra.run.smithi161.stdout:72 files removed 2023-10-31T12:00:15.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw 2023-10-31T12:00:16.659 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 234 kB/s | 81 kB 00:00 2023-10-31T12:00:17.184 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 20 kB/s | 9.4 kB 00:00 2023-10-31T12:00:17.776 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 3.5 kB/s | 1.9 kB 00:00 2023-10-31T12:00:18.162 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-10-31T12:00:19.488 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-10-31T12:00:20.895 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 7.3 MB/s | 7.9 MB 00:01 2023-10-31T12:00:24.005 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 18 MB 00:02 2023-10-31T12:00:35.563 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 719 kB/s | 5.8 MB 00:08 2023-10-31T12:00:37.088 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-10-31T12:00:39.234 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 19 MB 00:02 2023-10-31T12:00:43.856 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2023-10-31T12:00:46.241 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:00:46.253 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:00:46.253 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:00:46.254 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:00:46.254 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:00:46.254 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 8.4 M 2023-10-31T12:00:46.254 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2023-10-31T12:00:46.255 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 3.1 M 2023-10-31T12:00:46.255 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 3.0 M 2023-10-31T12:00:46.255 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:00:46.255 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options 2023-10-31T12:00:46.255 INFO:teuthology.orchestra.run.smithi161.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-31T12:00:46.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 6.6 M 2023-10-31T12:00:46.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 17 M 2023-10-31T12:00:46.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 24 k 2023-10-31T12:00:46.256 INFO:teuthology.orchestra.run.smithi161.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-31T12:00:46.256 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-31T12:00:46.257 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-31T12:00:46.257 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-31T12:00:46.257 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 727 k 2023-10-31T12:00:46.257 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-31T12:00:46.257 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-31T12:00:46.258 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-31T12:00:46.258 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 469 k 2023-10-31T12:00:46.258 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-31T12:00:46.258 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 4.5 M 2023-10-31T12:00:46.258 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-31T12:00:46.259 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-31T12:00:46.259 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-31T12:00:46.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse 2023-10-31T12:00:46.259 INFO:teuthology.orchestra.run.smithi161.stdout: x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 45 k 2023-10-31T12:00:46.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 127 k 2023-10-31T12:00:46.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 162 k 2023-10-31T12:00:46.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-31T12:00:46.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 320 k 2023-10-31T12:00:46.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 298 k 2023-10-31T12:00:46.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 99 k 2023-10-31T12:00:46.260 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-31T12:00:46.261 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-31T12:00:46.261 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:00:46.261 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:00:46.261 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:00:46.262 INFO:teuthology.orchestra.run.smithi161.stdout:Install 28 Packages 2023-10-31T12:00:46.262 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 2 Packages 2023-10-31T12:00:46.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:00:46.262 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 54 M 2023-10-31T12:00:46.263 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:00:48.107 INFO:teuthology.orchestra.run.smithi161.stdout:(1/30): ceph-base-18.0.0-6792.ge83bf482.el9.x86 4.7 MB/s | 6.6 MB 00:01 2023-10-31T12:00:48.157 INFO:teuthology.orchestra.run.smithi161.stdout:(2/30): ceph-selinux-18.0.0-6792.ge83bf482.el9. 485 kB/s | 24 kB 00:00 2023-10-31T12:00:48.266 INFO:teuthology.orchestra.run.smithi161.stdout:(3/30): libcephfs2-18.0.0-6792.ge83bf482.el9.x8 6.6 MB/s | 727 kB 00:00 2023-10-31T12:00:48.333 INFO:teuthology.orchestra.run.smithi161.stdout:(4/30): libradosstriper1-18.0.0-6792.ge83bf482. 6.9 MB/s | 469 kB 00:00 2023-10-31T12:00:48.741 INFO:teuthology.orchestra.run.smithi161.stdout:(5/30): ceph-radosgw-18.0.0-6792.ge83bf482.el9. 4.1 MB/s | 8.4 MB 00:02 2023-10-31T12:00:48.800 INFO:teuthology.orchestra.run.smithi161.stdout:(6/30): librgw2-18.0.0-6792.ge83bf482.el9.x86_6 9.6 MB/s | 4.5 MB 00:00 2023-10-31T12:00:48.834 INFO:teuthology.orchestra.run.smithi161.stdout:(7/30): python3-ceph-argparse-18.0.0-6792.ge83b 490 kB/s | 45 kB 00:00 2023-10-31T12:00:48.860 INFO:teuthology.orchestra.run.smithi161.stdout:(8/30): python3-ceph-common-18.0.0-6792.ge83bf4 2.1 MB/s | 127 kB 00:00 2023-10-31T12:00:48.893 INFO:teuthology.orchestra.run.smithi161.stdout:(9/30): python3-cephfs-18.0.0-6792.ge83bf482.el 2.7 MB/s | 162 kB 00:00 2023-10-31T12:00:48.918 INFO:teuthology.orchestra.run.smithi161.stdout:(10/30): python3-rados-18.0.0-6792.ge83bf482.el 5.3 MB/s | 320 kB 00:00 2023-10-31T12:00:48.952 INFO:teuthology.orchestra.run.smithi161.stdout:(11/30): python3-rbd-18.0.0-6792.ge83bf482.el9. 5.0 MB/s | 298 kB 00:00 2023-10-31T12:00:48.985 INFO:teuthology.orchestra.run.smithi161.stdout:(12/30): python3-rgw-18.0.0-6792.ge83bf482.el9. 1.4 MB/s | 99 kB 00:00 2023-10-31T12:00:49.102 INFO:teuthology.orchestra.run.smithi161.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2023-10-31T12:00:49.136 INFO:teuthology.orchestra.run.smithi161.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-10-31T12:00:49.177 INFO:teuthology.orchestra.run.smithi161.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-10-31T12:00:49.211 INFO:teuthology.orchestra.run.smithi161.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-10-31T12:00:49.237 INFO:teuthology.orchestra.run.smithi161.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 706 kB/s | 41 kB 00:00 2023-10-31T12:00:49.529 INFO:teuthology.orchestra.run.smithi161.stdout:(18/30): ceph-common-18.0.0-6792.ge83bf482.el9. 6.1 MB/s | 17 MB 00:02 2023-10-31T12:00:49.612 INFO:teuthology.orchestra.run.smithi161.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 295 kB/s | 111 kB 00:00 2023-10-31T12:00:49.637 INFO:teuthology.orchestra.run.smithi161.stdout:(20/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 106 kB/s | 45 kB 00:00 2023-10-31T12:00:49.820 INFO:teuthology.orchestra.run.smithi161.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 134 kB/s | 24 kB 00:00 2023-10-31T12:00:49.862 INFO:teuthology.orchestra.run.smithi161.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 927 kB/s | 309 kB 00:00 2023-10-31T12:00:49.904 INFO:teuthology.orchestra.run.smithi161.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2023-10-31T12:00:49.937 INFO:teuthology.orchestra.run.smithi161.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 893 kB/s | 67 kB 00:00 2023-10-31T12:00:50.021 INFO:teuthology.orchestra.run.smithi161.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2023-10-31T12:00:50.172 INFO:teuthology.orchestra.run.smithi161.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.1 MB/s | 837 kB 00:00 2023-10-31T12:00:50.439 INFO:teuthology.orchestra.run.smithi161.stdout:(27/30): librados2-18.0.0-6792.ge83bf482.el9.x8 12 MB/s | 3.1 MB 00:00 2023-10-31T12:00:50.614 INFO:teuthology.orchestra.run.smithi161.stdout:(28/30): librbd1-18.0.0-6792.ge83bf482.el9.x86_ 17 MB/s | 3.0 MB 00:00 2023-10-31T12:00:50.664 INFO:teuthology.orchestra.run.smithi161.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2023-10-31T12:00:51.123 INFO:teuthology.orchestra.run.smithi161.stdout:(30/30): libarrow-9.0.0-9.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2023-10-31T12:00:51.128 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:00:51.128 INFO:teuthology.orchestra.run.smithi161.stdout:Total 11 MB/s | 54 MB 00:04 2023-10-31T12:00:51.140 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:00:51.175 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:00:51.175 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:00:51.635 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:00:51.636 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:00:53.154 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:00:53.512 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-31T12:00:53.591 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 2/32 2023-10-31T12:00:53.683 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 2/32 2023-10-31T12:00:53.771 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 3/32 2023-10-31T12:00:53.801 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 4/32 2023-10-31T12:00:53.847 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 4/32 2023-10-31T12:00:53.888 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-31T12:00:53.922 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-31T12:00:53.960 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-31T12:00:54.013 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:18.0.0-6792.ge83bf482.el9. 8/32 2023-10-31T12:00:54.080 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 9/32 2023-10-31T12:00:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_6 10/32 2023-10-31T12:00:54.193 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_6 10/32 2023-10-31T12:00:54.228 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-31T12:00:54.335 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-31T12:00:54.371 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-31T12:00:54.420 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-31T12:00:54.452 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-31T12:00:54.479 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-31T12:00:54.506 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-31T12:00:54.546 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-31T12:00:54.864 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-31T12:00:54.893 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 20/32 2023-10-31T12:00:54.965 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 20/32 2023-10-31T12:00:54.997 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 21/32 2023-10-31T12:00:55.034 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-31T12:00:55.467 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-31T12:00:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 24/32 2023-10-31T12:00:55.567 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 24/32 2023-10-31T12:00:55.642 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 25/32 2023-10-31T12:00:55.673 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:18.0.0-6792.ge83bf482.el9.x8 26/32 2023-10-31T12:00:57.717 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 27/32 2023-10-31T12:00:57.760 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 27/32 2023-10-31T12:00:58.493 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 27/32 2023-10-31T12:00:58.549 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 28/32 2023-10-31T12:00:58.638 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 28/32 2023-10-31T12:00:58.638 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-31T12:00:58.639 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-31T12:00:58.639 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:00:58.692 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 29/32 2023-10-31T12:01:08.292 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 29/32 2023-10-31T12:01:08.293 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2023-10-31T12:01:08.293 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2023-10-31T12:01:08.293 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2023-10-31T12:01:08.293 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2023-10-31T12:01:08.294 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2023-10-31T12:01:08.294 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2023-10-31T12:01:08.294 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2023-10-31T12:01:08.294 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:09.123 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 30/32 2023-10-31T12:01:09.148 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 30/32 2023-10-31T12:01:09.148 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:01:09.148 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-31T12:01:09.149 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-31T12:01:09.149 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-31T12:01:09.149 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:09.150 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-31T12:01:09.225 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-31T12:01:09.225 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T12:01:10.366 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T12:01:10.367 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 1/32 2023-10-31T12:01:10.367 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 2/32 2023-10-31T12:01:10.367 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 3/32 2023-10-31T12:01:10.367 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 4/32 2023-10-31T12:01:10.367 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 5/32 2023-10-31T12:01:10.368 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_6 6/32 2023-10-31T12:01:10.368 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 7/32 2023-10-31T12:01:10.368 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6792.ge83bf482.el9. 8/32 2023-10-31T12:01:10.368 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:18.0.0-6792.ge83bf482.el9.x8 9/32 2023-10-31T12:01:10.368 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 10/32 2023-10-31T12:01:10.369 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 11/32 2023-10-31T12:01:10.369 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 12/32 2023-10-31T12:01:10.369 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 13/32 2023-10-31T12:01:10.369 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-31T12:01:10.369 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-31T12:01:10.370 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-31T12:01:10.370 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-31T12:01:10.370 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-31T12:01:10.370 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-31T12:01:10.370 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-31T12:01:10.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-31T12:01:10.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-31T12:01:10.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-31T12:01:10.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-31T12:01:10.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-31T12:01:10.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-31T12:01:10.372 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-31T12:01:10.372 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-31T12:01:10.372 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 29/32 2023-10-31T12:01:10.372 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-31T12:01:10.372 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 31/32 2023-10-31T12:01:11.310 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T12:01:11.310 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:11.310 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2023-10-31T12:01:11.311 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.311 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.311 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:01:11.311 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-31T12:01:11.312 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.312 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.312 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.312 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.312 INFO:teuthology.orchestra.run.smithi161.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-31T12:01:11.313 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-31T12:01:11.313 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-31T12:01:11.313 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-31T12:01:11.313 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.313 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-31T12:01:11.314 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-31T12:01:11.314 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-31T12:01:11.314 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.314 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-31T12:01:11.315 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.315 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-31T12:01:11.315 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-31T12:01:11.315 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-31T12:01:11.315 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-31T12:01:11.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.317 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.317 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:11.317 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2023-10-31T12:01:11.317 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-31T12:01:11.317 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:11.318 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:01:11.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-test 2023-10-31T12:01:12.101 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:29 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:01:12.243 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:01:12.244 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:01:12.245 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:01:12.245 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:01:12.245 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:01:12.245 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 117 M 2023-10-31T12:01:12.245 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:01:12.246 INFO:teuthology.orchestra.run.smithi161.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-31T12:01:12.246 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-31T12:01:12.246 INFO:teuthology.orchestra.run.smithi161.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-31T12:01:12.246 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 161 k 2023-10-31T12:01:12.246 INFO:teuthology.orchestra.run.smithi161.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-31T12:01:12.247 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-31T12:01:12.247 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-31T12:01:12.247 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-31T12:01:12.247 INFO:teuthology.orchestra.run.smithi161.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-31T12:01:12.248 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:12.248 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:01:12.248 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:01:12.248 INFO:teuthology.orchestra.run.smithi161.stdout:Install 10 Packages 2023-10-31T12:01:12.249 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:12.249 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 120 M 2023-10-31T12:01:12.249 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.1 G 2023-10-31T12:01:12.250 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:01:12.872 INFO:teuthology.orchestra.run.smithi161.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.4 MB/s | 418 kB 00:00 2023-10-31T12:01:12.906 INFO:teuthology.orchestra.run.smithi161.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.6 MB/s | 186 kB 00:00 2023-10-31T12:01:12.939 INFO:teuthology.orchestra.run.smithi161.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2023-10-31T12:01:12.989 INFO:teuthology.orchestra.run.smithi161.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2023-10-31T12:01:13.014 INFO:teuthology.orchestra.run.smithi161.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-10-31T12:01:13.548 INFO:teuthology.orchestra.run.smithi161.stdout:(6/10): libcephsqlite-18.0.0-6792.ge83bf482.el9 191 kB/s | 161 kB 00:00 2023-10-31T12:01:13.690 INFO:teuthology.orchestra.run.smithi161.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 277 kB/s | 187 kB 00:00 2023-10-31T12:01:14.115 INFO:teuthology.orchestra.run.smithi161.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 287 kB/s | 122 kB 00:00 2023-10-31T12:01:14.782 INFO:teuthology.orchestra.run.smithi161.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.0 MB/s | 1.3 MB 00:01 2023-10-31T12:01:27.023 INFO:teuthology.orchestra.run.smithi161.stdout:(10/10): ceph-test-18.0.0-6792.ge83bf482.el9.x8 8.2 MB/s | 117 MB 00:14 2023-10-31T12:01:27.027 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:01:27.028 INFO:teuthology.orchestra.run.smithi161.stdout:Total 8.1 MB/s | 120 MB 00:14 2023-10-31T12:01:27.041 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:01:27.065 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:01:27.065 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:01:27.886 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:01:27.887 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:01:29.003 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:01:29.102 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-31T12:01:29.139 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-31T12:01:29.183 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-31T12:01:29.233 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-31T12:01:29.265 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-31T12:01:29.297 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-31T12:01:29.326 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-31T12:01:29.375 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-31T12:01:29.411 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 9/10 2023-10-31T12:01:45.708 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 9/10 2023-10-31T12:01:45.800 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 10/10 2023-10-31T12:01:53.756 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 10/10 2023-10-31T12:01:53.757 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 1/10 2023-10-31T12:01:53.757 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 2/10 2023-10-31T12:01:53.757 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-31T12:01:53.757 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-31T12:01:53.758 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-31T12:01:53.758 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-31T12:01:53.758 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-31T12:01:53.758 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-31T12:01:53.759 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-31T12:01:54.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-31T12:01:54.211 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:54.212 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:01:54.212 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:54.212 INFO:teuthology.orchestra.run.smithi161.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-31T12:01:54.212 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-15.el9.x86_64 2023-10-31T12:01:54.212 INFO:teuthology.orchestra.run.smithi161.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-31T12:01:54.213 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:01:54.213 INFO:teuthology.orchestra.run.smithi161.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-31T12:01:54.213 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-31T12:01:54.213 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-31T12:01:54.213 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-31T12:01:54.214 INFO:teuthology.orchestra.run.smithi161.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-31T12:01:54.214 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:54.214 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:01:54.543 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph 2023-10-31T12:01:55.097 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:12 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:01:55.252 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:01:55.254 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2023-10-31T12:01:55.255 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:01:55.255 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2023-10-31T12:01:55.255 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:01:55.255 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 6.5 k 2023-10-31T12:01:55.255 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:01:55.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 2.0 M 2023-10-31T12:01:55.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 1.5 M 2023-10-31T12:01:55.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 242 k 2023-10-31T12:01:55.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 8.8 M 2023-10-31T12:01:55.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 116 M 2023-10-31T12:01:55.257 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-31T12:01:55.257 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-31T12:01:55.257 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-31T12:01:55.257 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-31T12:01:55.257 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-31T12:01:55.257 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-31T12:01:55.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-31T12:01:55.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-31T12:01:55.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-31T12:01:55.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-31T12:01:55.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-31T12:01:55.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-31T12:01:55.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-31T12:01:55.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-31T12:01:55.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-31T12:01:55.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-31T12:01:55.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-31T12:01:55.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-31T12:01:55.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-31T12:01:55.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-31T12:01:55.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-31T12:01:55.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-31T12:01:55.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-31T12:01:55.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-31T12:01:55.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-31T12:01:55.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-31T12:01:55.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-31T12:01:55.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:55.262 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:01:55.262 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2023-10-31T12:01:55.262 INFO:teuthology.orchestra.run.smithi161.stdout:Install 33 Packages 2023-10-31T12:01:55.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:01:55.263 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 133 M 2023-10-31T12:01:55.263 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.4 G 2023-10-31T12:01:55.263 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:01:55.857 INFO:teuthology.orchestra.run.smithi161.stdout:(1/33): ceph-18.0.0-6792.ge83bf482.el9.x86_64.r 34 kB/s | 6.5 kB 00:00 2023-10-31T12:01:56.758 INFO:teuthology.orchestra.run.smithi161.stdout:(2/33): ceph-mon-18.0.0-6792.ge83bf482.el9.x86_ 9.8 MB/s | 8.8 MB 00:00 2023-10-31T12:01:57.473 INFO:teuthology.orchestra.run.smithi161.stdout:(3/33): ceph-mgr-18.0.0-6792.ge83bf482.el9.x86_ 823 kB/s | 1.5 MB 00:01 2023-10-31T12:01:57.758 INFO:teuthology.orchestra.run.smithi161.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6792.ge83b 851 kB/s | 242 kB 00:00 2023-10-31T12:01:57.859 INFO:teuthology.orchestra.run.smithi161.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-10-31T12:01:57.901 INFO:teuthology.orchestra.run.smithi161.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2023-10-31T12:01:57.934 INFO:teuthology.orchestra.run.smithi161.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-31T12:01:57.976 INFO:teuthology.orchestra.run.smithi161.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-31T12:01:58.010 INFO:teuthology.orchestra.run.smithi161.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-31T12:01:58.044 INFO:teuthology.orchestra.run.smithi161.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-10-31T12:01:58.077 INFO:teuthology.orchestra.run.smithi161.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-31T12:01:58.111 INFO:teuthology.orchestra.run.smithi161.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-10-31T12:01:58.253 INFO:teuthology.orchestra.run.smithi161.stdout:(13/33): ceph-mds-18.0.0-6792.ge83bf482.el9.x86 790 kB/s | 2.0 MB 00:02 2023-10-31T12:01:58.437 INFO:teuthology.orchestra.run.smithi161.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 278 kB/s | 51 kB 00:00 2023-10-31T12:01:58.745 INFO:teuthology.orchestra.run.smithi161.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2023-10-31T12:01:58.979 INFO:teuthology.orchestra.run.smithi161.stdout:(16/33): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2023-10-31T12:01:59.004 INFO:teuthology.orchestra.run.smithi161.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 664 kB/s | 172 kB 00:00 2023-10-31T12:01:59.163 INFO:teuthology.orchestra.run.smithi161.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2023-10-31T12:01:59.238 INFO:teuthology.orchestra.run.smithi161.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 236 kB/s | 18 kB 00:00 2023-10-31T12:01:59.330 INFO:teuthology.orchestra.run.smithi161.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 253 kB/s | 23 kB 00:00 2023-10-31T12:01:59.430 INFO:teuthology.orchestra.run.smithi161.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2023-10-31T12:01:59.488 INFO:teuthology.orchestra.run.smithi161.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 704 kB/s | 358 kB 00:00 2023-10-31T12:01:59.514 INFO:teuthology.orchestra.run.smithi161.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2023-10-31T12:01:59.556 INFO:teuthology.orchestra.run.smithi161.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2023-10-31T12:01:59.606 INFO:teuthology.orchestra.run.smithi161.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 858 kB/s | 79 kB 00:00 2023-10-31T12:01:59.673 INFO:teuthology.orchestra.run.smithi161.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2023-10-31T12:01:59.690 INFO:teuthology.orchestra.run.smithi161.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2023-10-31T12:01:59.774 INFO:teuthology.orchestra.run.smithi161.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2023-10-31T12:01:59.799 INFO:teuthology.orchestra.run.smithi161.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 828 kB/s | 90 kB 00:00 2023-10-31T12:01:59.882 INFO:teuthology.orchestra.run.smithi161.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2023-10-31T12:01:59.949 INFO:teuthology.orchestra.run.smithi161.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2023-10-31T12:02:00.025 INFO:teuthology.orchestra.run.smithi161.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2023-10-31T12:02:01.983 INFO:teuthology.orchestra.run.smithi161.stdout:(33/33): ceph-osd-18.0.0-6792.ge83bf482.el9.x86 22 MB/s | 116 MB 00:05 2023-10-31T12:02:01.987 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:02:01.987 INFO:teuthology.orchestra.run.smithi161.stdout:Total 20 MB/s | 133 MB 00:06 2023-10-31T12:02:02.020 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:02:02.059 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:02:02.059 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:02:03.035 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:02:03.035 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:02:04.881 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:02:04.944 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-31T12:02:04.976 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-31T12:02:05.025 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-31T12:02:05.117 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-31T12:02:05.203 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-31T12:02:05.245 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-31T12:02:05.282 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-31T12:02:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-31T12:02:05.404 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-31T12:02:05.443 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-31T12:02:05.508 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-31T12:02:05.572 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-31T12:02:05.765 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-31T12:02:05.814 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-31T12:02:05.861 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-31T12:02:05.895 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-31T12:02:05.925 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-31T12:02:06.002 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-31T12:02:06.044 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-31T12:02:06.077 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-31T12:02:06.119 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-31T12:02:06.153 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-31T12:02:06.242 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-31T12:02:06.293 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-31T12:02:06.373 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-31T12:02:06.518 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-31T12:02:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-31T12:02:06.809 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6792.ge83bf482.el9. 28/33 2023-10-31T12:02:06.847 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 29/33 2023-10-31T12:02:06.873 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 29/33 2023-10-31T12:02:06.873 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:02:06.873 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-31T12:02:06.873 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-31T12:02:06.873 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-31T12:02:06.874 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:25.229 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 30/33 2023-10-31T12:02:25.256 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 30/33 2023-10-31T12:02:25.256 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:02:25.257 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-31T12:02:25.257 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-31T12:02:25.257 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-31T12:02:25.257 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:26.181 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 31/33 2023-10-31T12:02:26.206 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 31/33 2023-10-31T12:02:26.206 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:02:26.207 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-31T12:02:26.207 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-31T12:02:26.207 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-31T12:02:26.207 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:26.406 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 32/33 2023-10-31T12:02:26.431 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 32/33 2023-10-31T12:02:26.431 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:02:26.431 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-31T12:02:26.432 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-31T12:02:26.432 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-31T12:02:26.432 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:26.495 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 33/33 2023-10-31T12:02:36.128 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 33/33 2023-10-31T12:02:36.128 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 1/33 2023-10-31T12:02:36.128 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 2/33 2023-10-31T12:02:36.129 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 3/33 2023-10-31T12:02:36.129 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 4/33 2023-10-31T12:02:36.129 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 5/33 2023-10-31T12:02:36.129 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6792.ge83bf482.el9. 6/33 2023-10-31T12:02:36.129 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-31T12:02:36.130 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-31T12:02:36.130 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-31T12:02:36.130 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-31T12:02:36.130 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-31T12:02:36.130 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-31T12:02:36.131 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-31T12:02:36.131 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-31T12:02:36.131 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-31T12:02:36.131 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-31T12:02:36.131 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-31T12:02:36.132 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-31T12:02:36.132 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-31T12:02:36.132 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-31T12:02:36.132 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-31T12:02:36.132 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-31T12:02:36.133 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-31T12:02:36.133 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-31T12:02:36.133 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-31T12:02:36.133 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-31T12:02:36.133 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-31T12:02:36.134 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-31T12:02:36.134 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-31T12:02:36.135 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-31T12:02:36.135 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-31T12:02:36.135 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-31T12:02:36.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-31T12:02:36.916 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:36.916 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:02:36.916 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:02:36.917 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:02:36.917 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:02:36.917 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:02:36.917 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:02:36.918 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:02:36.918 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-31T12:02:36.918 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-31T12:02:36.918 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-31T12:02:36.918 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-31T12:02:36.919 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-31T12:02:36.919 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-31T12:02:36.919 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-31T12:02:36.919 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-31T12:02:36.919 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-31T12:02:36.920 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-31T12:02:36.920 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-31T12:02:36.920 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-31T12:02:36.920 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-31T12:02:36.921 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-31T12:02:36.921 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-31T12:02:36.921 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2023-10-31T12:02:36.921 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-31T12:02:36.921 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-31T12:02:36.921 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-31T12:02:36.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-31T12:02:36.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-31T12:02:36.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-31T12:02:36.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-31T12:02:36.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-31T12:02:36.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-31T12:02:36.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-31T12:02:36.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-31T12:02:36.924 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:36.924 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:37.169 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-base 2023-10-31T12:02:37.712 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:54 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:37.805 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:02:37.854 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:37.855 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:02:37.855 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:37.922 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install cephadm 2023-10-31T12:02:38.463 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:55 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:38.604 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:38.605 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:38.605 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:02:38.605 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:38.605 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:02:38.606 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 270 k 2023-10-31T12:02:38.606 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:38.606 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:02:38.606 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:38.606 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:02:38.607 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:38.607 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 270 k 2023-10-31T12:02:38.607 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 265 k 2023-10-31T12:02:38.607 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:02:38.839 INFO:teuthology.orchestra.run.smithi161.stdout:cephadm-18.0.0-6792.ge83bf482.el9.noarch.rpm 1.1 MB/s | 270 kB 00:00 2023-10-31T12:02:38.840 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:02:38.840 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.1 MB/s | 270 kB 00:00 2023-10-31T12:02:38.841 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:02:38.845 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:02:38.846 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:02:38.848 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:02:38.849 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:02:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:02:39.303 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:02:39.377 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:02:39.745 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:02:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:02:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:40.094 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:02:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:02:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:40.094 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:40.360 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-immutable-object-cache 2023-10-31T12:02:40.916 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:57 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:41.062 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:41.063 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:41.063 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2023-10-31T12:02:41.063 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:41.063 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:02:41.063 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 144 k 2023-10-31T12:02:41.064 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:41.064 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:02:41.064 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:41.064 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:02:41.064 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:41.065 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 144 k 2023-10-31T12:02:41.065 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 437 k 2023-10-31T12:02:41.065 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:02:41.311 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-immutable-object-cache-18.0.0-6792.ge83bf4 585 kB/s | 144 kB 00:00 2023-10-31T12:02:41.312 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:02:41.312 INFO:teuthology.orchestra.run.smithi161.stdout:Total 581 kB/s | 144 kB 00:00 2023-10-31T12:02:41.312 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:02:41.319 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:02:41.319 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:02:41.371 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:02:41.371 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:02:41.516 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:02:41.587 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482. 1/1 2023-10-31T12:02:41.611 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482. 1/1 2023-10-31T12:02:41.611 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:02:41.611 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-31T12:02:41.611 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:42.621 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482. 1/1 2023-10-31T12:02:42.621 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:42.621 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:02:42.622 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:02:42.622 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:42.622 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:42.854 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr 2023-10-31T12:02:43.408 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:00 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:43.500 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:02:43.549 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:43.550 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:02:43.550 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:43.610 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-dashboard 2023-10-31T12:02:44.150 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:01 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:44.290 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:44.290 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:44.291 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:02:44.291 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:44.291 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:02:44.291 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 1.8 M 2023-10-31T12:02:44.291 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:02:44.292 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 23 k 2023-10-31T12:02:44.292 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 15 k 2023-10-31T12:02:44.292 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-31T12:02:44.292 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-31T12:02:44.292 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-31T12:02:44.293 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2023-10-31T12:02:44.293 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-31T12:02:44.293 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:44.293 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:02:44.293 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:02:44.294 INFO:teuthology.orchestra.run.smithi161.stdout:Install 7 Packages 2023-10-31T12:02:44.294 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 2.1 M 2023-10-31T12:02:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 8.6 M 2023-10-31T12:02:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:02:44.681 INFO:teuthology.orchestra.run.smithi161.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6792.ge83 119 kB/s | 23 kB 00:00 2023-10-31T12:02:44.731 INFO:teuthology.orchestra.run.smithi161.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6792.ge83b 62 kB/s | 15 kB 00:00 2023-10-31T12:02:44.832 INFO:teuthology.orchestra.run.smithi161.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 60 kB/s | 9.0 kB 00:00 2023-10-31T12:02:44.957 INFO:teuthology.orchestra.run.smithi161.stdout:(4/7): ceph-mgr-dashboard-18.0.0-6792.ge83bf482 3.7 MB/s | 1.8 MB 00:00 2023-10-31T12:02:44.982 INFO:teuthology.orchestra.run.smithi161.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 163 kB/s | 41 kB 00:00 2023-10-31T12:02:45.007 INFO:teuthology.orchestra.run.smithi161.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 175 kB/s | 31 kB 00:00 2023-10-31T12:02:45.158 INFO:teuthology.orchestra.run.smithi161.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 941 kB/s | 188 kB 00:00 2023-10-31T12:02:45.159 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:02:45.160 INFO:teuthology.orchestra.run.smithi161.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-10-31T12:02:45.160 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:02:45.167 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:02:45.168 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:02:45.229 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:02:45.229 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:02:45.526 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:02:45.592 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-31T12:02:45.626 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-31T12:02:45.672 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-31T12:02:45.704 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-31T12:02:45.733 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6792.ge83bf482.el9.n 5/7 2023-10-31T12:02:45.967 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6792.ge83bf482.el9. 6/7 2023-10-31T12:02:45.994 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarc 7/7 2023-10-31T12:02:46.560 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarc 7/7 2023-10-31T12:02:46.561 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6792.ge83bf482.el9. 1/7 2023-10-31T12:02:46.561 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarc 2/7 2023-10-31T12:02:46.561 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6792.ge83bf482.el9.n 3/7 2023-10-31T12:02:46.561 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-31T12:02:46.562 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-31T12:02:46.562 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-31T12:02:46.883 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-31T12:02:46.884 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:46.884 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:02:46.884 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:02:46.884 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:02:46.885 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:02:46.885 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-31T12:02:46.885 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-31T12:02:46.885 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-31T12:02:46.885 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-31T12:02:46.886 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:46.886 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:47.098 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-31T12:02:47.640 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:04 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:47.782 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:47.783 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2023-10-31T12:02:47.784 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:02:47.784 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2023-10-31T12:02:47.784 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:02:47.784 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 7.4 M 2023-10-31T12:02:47.784 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:02:47.785 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-31T12:02:47.785 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-31T12:02:47.785 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-31T12:02:47.785 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-31T12:02:47.785 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-31T12:02:47.786 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-31T12:02:47.786 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-31T12:02:47.786 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-31T12:02:47.786 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-31T12:02:47.786 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-31T12:02:47.787 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-31T12:02:47.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:47.787 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:02:47.787 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2023-10-31T12:02:47.787 INFO:teuthology.orchestra.run.smithi161.stdout:Install 12 Packages 2023-10-31T12:02:47.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:47.788 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 37 M 2023-10-31T12:02:47.788 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 209 M 2023-10-31T12:02:47.789 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:02:47.906 INFO:teuthology.orchestra.run.smithi161.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 265 kB/s | 32 kB 00:00 2023-10-31T12:02:47.939 INFO:teuthology.orchestra.run.smithi161.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 517 kB/s | 17 kB 00:00 2023-10-31T12:02:47.973 INFO:teuthology.orchestra.run.smithi161.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-31T12:02:48.056 INFO:teuthology.orchestra.run.smithi161.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-10-31T12:02:48.115 INFO:teuthology.orchestra.run.smithi161.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.5 MB/s | 205 kB 00:00 2023-10-31T12:02:48.182 INFO:teuthology.orchestra.run.smithi161.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-10-31T12:02:48.257 INFO:teuthology.orchestra.run.smithi161.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-10-31T12:02:48.283 INFO:teuthology.orchestra.run.smithi161.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-31T12:02:48.483 INFO:teuthology.orchestra.run.smithi161.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 70 MB/s | 16 MB 00:00 2023-10-31T12:02:48.516 INFO:teuthology.orchestra.run.smithi161.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2023-10-31T12:02:48.541 INFO:teuthology.orchestra.run.smithi161.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-10-31T12:02:50.058 INFO:teuthology.orchestra.run.smithi161.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 3.3 MB/s | 7.4 MB 00:02 2023-10-31T12:02:50.062 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:02:50.062 INFO:teuthology.orchestra.run.smithi161.stdout:Total 16 MB/s | 37 MB 00:02 2023-10-31T12:02:50.063 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:02:50.087 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:02:50.088 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:02:50.523 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:02:50.524 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:02:51.220 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:02:51.289 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-31T12:02:51.329 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-31T12:02:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-31T12:02:51.466 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-31T12:02:51.644 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-31T12:02:51.673 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-31T12:02:51.776 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-31T12:02:52.427 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-31T12:02:52.535 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-31T12:02:53.785 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-31T12:02:54.825 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-31T12:02:54.855 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf 12/12 2023-10-31T12:02:57.442 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf 12/12 2023-10-31T12:02:57.442 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf 1/12 2023-10-31T12:02:57.443 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-31T12:02:57.443 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-31T12:02:57.443 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-31T12:02:57.443 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-31T12:02:57.443 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-31T12:02:57.444 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-31T12:02:57.444 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-31T12:02:57.444 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-31T12:02:57.444 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-31T12:02:57.445 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-31T12:02:57.932 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-31T12:02:57.933 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:57.933 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:02:57.933 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:02:57.933 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-31T12:02:57.933 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-31T12:02:57.934 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-31T12:02:57.934 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-31T12:02:57.934 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-31T12:02:57.934 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-31T12:02:57.934 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-31T12:02:57.934 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-31T12:02:57.935 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-31T12:02:57.935 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-31T12:02:57.935 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-31T12:02:57.935 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:57.935 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:02:58.285 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-rook 2023-10-31T12:02:58.841 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:15 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:02:58.984 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:02:58.985 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2023-10-31T12:02:58.985 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:02:58.985 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2023-10-31T12:02:58.985 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:02:58.986 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 48 k 2023-10-31T12:02:58.986 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:02:58.986 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-31T12:02:58.986 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-31T12:02:58.987 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-31T12:02:58.987 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-31T12:02:58.987 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-31T12:02:58.987 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-31T12:02:58.987 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-31T12:02:58.987 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-31T12:02:58.988 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-31T12:02:58.988 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-31T12:02:58.988 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-31T12:02:58.988 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-31T12:02:58.988 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:58.989 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:02:58.989 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2023-10-31T12:02:58.989 INFO:teuthology.orchestra.run.smithi161.stdout:Install 13 Packages 2023-10-31T12:02:58.989 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:02:58.990 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 2.0 M 2023-10-31T12:02:58.990 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 26 M 2023-10-31T12:02:58.991 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:02:59.350 INFO:teuthology.orchestra.run.smithi161.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 24 kB 00:00 2023-10-31T12:02:59.367 INFO:teuthology.orchestra.run.smithi161.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 18 kB 00:00 2023-10-31T12:02:59.409 INFO:teuthology.orchestra.run.smithi161.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2023-10-31T12:02:59.434 INFO:teuthology.orchestra.run.smithi161.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-31T12:02:59.459 INFO:teuthology.orchestra.run.smithi161.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-31T12:02:59.476 INFO:teuthology.orchestra.run.smithi161.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2023-10-31T12:02:59.610 INFO:teuthology.orchestra.run.smithi161.stdout:(7/13): ceph-mgr-rook-18.0.0-6792.ge83bf482.el9 125 kB/s | 48 kB 00:00 2023-10-31T12:02:59.726 INFO:teuthology.orchestra.run.smithi161.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2023-10-31T12:02:59.768 INFO:teuthology.orchestra.run.smithi161.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2023-10-31T12:02:59.902 INFO:teuthology.orchestra.run.smithi161.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2023-10-31T12:02:59.969 INFO:teuthology.orchestra.run.smithi161.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 549 kB/s | 197 kB 00:00 2023-10-31T12:03:00.019 INFO:teuthology.orchestra.run.smithi161.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2023-10-31T12:03:00.086 INFO:teuthology.orchestra.run.smithi161.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2023-10-31T12:03:00.090 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:00.091 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-10-31T12:03:00.091 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:00.104 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:00.104 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:00.202 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:00.204 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:00.885 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-31T12:03:00.937 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-31T12:03:01.001 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-31T12:03:01.038 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-31T12:03:01.080 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-31T12:03:01.181 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-31T12:03:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-31T12:03:01.340 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-31T12:03:02.069 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-31T12:03:02.136 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-31T12:03:02.179 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-31T12:03:02.224 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-31T12:03:02.248 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 13/13 2023-10-31T12:03:03.052 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 13/13 2023-10-31T12:03:03.052 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 1/13 2023-10-31T12:03:03.053 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-31T12:03:03.053 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-31T12:03:03.053 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-31T12:03:03.053 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-31T12:03:03.054 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-31T12:03:03.054 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-31T12:03:03.054 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-31T12:03:03.054 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-31T12:03:03.054 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-31T12:03:03.055 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-31T12:03:03.055 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-31T12:03:03.429 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-31T12:03:03.430 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:03.430 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:03.430 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:03:03.430 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-31T12:03:03.430 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-31T12:03:03.431 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-31T12:03:03.431 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-31T12:03:03.431 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-31T12:03:03.431 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-31T12:03:03.432 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-31T12:03:03.432 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-31T12:03:03.432 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-31T12:03:03.432 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-31T12:03:03.432 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-31T12:03:03.433 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-31T12:03:03.433 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:03.433 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:03.687 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-cephadm 2023-10-31T12:03:04.232 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:21 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:04.375 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:04.376 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2023-10-31T12:03:04.376 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:03:04.376 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2023-10-31T12:03:04.377 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:04.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 132 k 2023-10-31T12:03:04.377 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:03:04.377 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-31T12:03:04.377 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-31T12:03:04.378 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-31T12:03:04.378 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-31T12:03:04.378 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-31T12:03:04.378 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:04.378 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:04.379 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2023-10-31T12:03:04.379 INFO:teuthology.orchestra.run.smithi161.stdout:Install 6 Packages 2023-10-31T12:03:04.379 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:04.380 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 6.8 M 2023-10-31T12:03:04.380 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 33 M 2023-10-31T12:03:04.380 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:04.904 INFO:teuthology.orchestra.run.smithi161.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-31T12:03:05.013 INFO:teuthology.orchestra.run.smithi161.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-10-31T12:03:05.080 INFO:teuthology.orchestra.run.smithi161.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6792.ge83bf482.e 405 kB/s | 132 kB 00:00 2023-10-31T12:03:05.572 INFO:teuthology.orchestra.run.smithi161.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 108 kB/s | 58 kB 00:00 2023-10-31T12:03:05.605 INFO:teuthology.orchestra.run.smithi161.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 97 kB/s | 51 kB 00:00 2023-10-31T12:03:06.047 INFO:teuthology.orchestra.run.smithi161.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 480 kB/s | 548 kB 00:01 2023-10-31T12:03:06.051 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:06.052 INFO:teuthology.orchestra.run.smithi161.stdout:Total 4.0 MB/s | 6.8 MB 00:01 2023-10-31T12:03:06.052 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:06.061 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:06.062 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:06.154 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:06.155 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:06.459 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-31T12:03:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-31T12:03:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-31T12:03:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-31T12:03:07.370 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-31T12:03:07.394 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 6/6 2023-10-31T12:03:08.214 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 6/6 2023-10-31T12:03:08.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/6 2023-10-31T12:03:08.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-31T12:03:08.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-31T12:03:08.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-31T12:03:08.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-31T12:03:08.532 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-31T12:03:08.533 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:08.533 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:08.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:03:08.533 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-31T12:03:08.534 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-31T12:03:08.534 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-31T12:03:08.534 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-31T12:03:08.534 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-31T12:03:08.534 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:08.535 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:08.772 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-fuse 2023-10-31T12:03:09.317 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:26 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:09.459 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:09.460 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:09.460 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:03:09.461 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:09.461 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:09.461 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 833 k 2023-10-31T12:03:09.461 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:03:09.461 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-31T12:03:09.462 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:09.462 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:09.462 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:09.462 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2023-10-31T12:03:09.462 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:09.463 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 911 k 2023-10-31T12:03:09.463 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 3.0 M 2023-10-31T12:03:09.463 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:09.582 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 651 kB/s | 79 kB 00:00 2023-10-31T12:03:09.782 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): ceph-fuse-18.0.0-6792.ge83bf482.el9.x86_ 2.5 MB/s | 833 kB 00:00 2023-10-31T12:03:09.784 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:09.784 INFO:teuthology.orchestra.run.smithi161.stdout:Total 2.8 MB/s | 911 kB 00:00 2023-10-31T12:03:09.784 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:09.805 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:09.805 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:09.882 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:09.882 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:10.071 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:10.222 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-31T12:03:10.290 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 2/2 2023-10-31T12:03:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 2/2 2023-10-31T12:03:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/2 2023-10-31T12:03:11.330 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-31T12:03:11.331 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:11.331 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:11.331 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-31T12:03:11.331 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:11.332 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:11.559 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-volume 2023-10-31T12:03:12.114 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:29 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:12.259 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:12.260 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:12.260 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:03:12.261 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:12.261 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:12.261 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 262 k 2023-10-31T12:03:12.261 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:12.261 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:12.262 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:12.262 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:03:12.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:12.262 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 262 k 2023-10-31T12:03:12.263 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.2 M 2023-10-31T12:03:12.263 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:12.486 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-volume-18.0.0-6792.ge83bf482.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-31T12:03:12.487 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:12.487 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-31T12:03:12.487 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:12.495 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:12.495 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:12.535 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:12.535 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:03:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:03:12.837 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:03:12.837 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-31T12:03:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:13.966 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:03:13.967 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:13.967 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:13.967 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:03:13.967 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:13.968 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:14.241 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados-devel 2023-10-31T12:03:14.801 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:31 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:14.943 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:14.944 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:14.944 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2023-10-31T12:03:14.945 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:14.945 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:14.945 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 127 k 2023-10-31T12:03:14.945 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:14.946 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:14.946 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:14.946 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:03:14.946 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:14.947 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 127 k 2023-10-31T12:03:14.947 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 456 k 2023-10-31T12:03:14.947 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:15.166 INFO:teuthology.orchestra.run.smithi161.stdout:librados-devel-18.0.0-6792.ge83bf482.el9.x86_64 572 kB/s | 127 kB 00:00 2023-10-31T12:03:15.167 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:15.167 INFO:teuthology.orchestra.run.smithi161.stdout:Total 569 kB/s | 127 kB 00:00 2023-10-31T12:03:15.168 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:15.173 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:15.173 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:15.230 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:15.230 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:15.376 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:15.481 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:15.974 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:16.273 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:16.273 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:16.273 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:16.274 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:03:16.274 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:16.274 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:16.534 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs2 2023-10-31T12:03:17.087 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:34 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:17.182 INFO:teuthology.orchestra.run.smithi161.stdout:Package libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:17.233 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:17.234 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:17.234 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:17.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs-devel 2023-10-31T12:03:17.848 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:34 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:17.992 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2023-10-31T12:03:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 31 k 2023-10-31T12:03:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:17.994 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:17.994 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:17.994 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:03:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:17.995 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 31 k 2023-10-31T12:03:17.995 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 139 k 2023-10-31T12:03:17.995 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:18.195 INFO:teuthology.orchestra.run.smithi161.stdout:libcephfs-devel-18.0.0-6792.ge83bf482.el9.x86_6 155 kB/s | 31 kB 00:00 2023-10-31T12:03:18.195 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:18.196 INFO:teuthology.orchestra.run.smithi161.stdout:Total 154 kB/s | 31 kB 00:00 2023-10-31T12:03:18.196 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:18.199 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:18.199 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:18.231 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:18.232 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:18.315 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:18.435 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:18.874 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:19.142 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:19.143 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:19.143 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:19.143 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:03:19.143 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:19.143 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:19.329 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados2 2023-10-31T12:03:19.872 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:36 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:19.965 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:20.015 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:20.016 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:20.016 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:20.081 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librbd1 2023-10-31T12:03:20.624 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:37 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:20.716 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:20.766 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:20.767 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:20.767 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:20.828 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rados 2023-10-31T12:03:21.374 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:38 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:21.466 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:21.516 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:21.517 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:21.517 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:21.582 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rgw 2023-10-31T12:03:22.123 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:39 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:22.215 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:22.264 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:22.265 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:22.265 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:22.336 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-cephfs 2023-10-31T12:03:22.884 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:39 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:22.977 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:23.028 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:23.029 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:23.029 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:23.099 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rbd 2023-10-31T12:03:23.644 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:40 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:23.737 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:03:23.787 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:23.787 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2023-10-31T12:03:23.788 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:23.854 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-fuse 2023-10-31T12:03:24.402 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:41 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:24.544 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:24.545 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:24.545 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2023-10-31T12:03:24.545 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:24.545 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:24.545 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 87 k 2023-10-31T12:03:24.546 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:24.546 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:24.546 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:24.546 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:03:24.547 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:24.547 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 87 k 2023-10-31T12:03:24.547 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 252 k 2023-10-31T12:03:24.547 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:24.761 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-fuse-18.0.0-6792.ge83bf482.el9.x86_64.rpm 402 kB/s | 87 kB 00:00 2023-10-31T12:03:24.762 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:24.762 INFO:teuthology.orchestra.run.smithi161.stdout:Total 400 kB/s | 87 kB 00:00 2023-10-31T12:03:24.762 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:24.768 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:24.769 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:24.825 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:24.825 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:24.978 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:25.080 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:25.494 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:25.802 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:25.802 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:25.802 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:25.802 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:03:25.803 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:25.803 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:26.066 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-mirror 2023-10-31T12:03:26.630 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:43 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:26.779 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:26.780 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:26.781 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2023-10-31T12:03:26.781 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:26.782 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:26.782 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 3.0 M 2023-10-31T12:03:26.782 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:26.782 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:26.783 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:26.783 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:03:26.783 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:26.783 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 3.0 M 2023-10-31T12:03:26.783 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 13 M 2023-10-31T12:03:26.784 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:27.160 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-mirror-18.0.0-6792.ge83bf482.el9.x86_64.rpm 7.9 MB/s | 3.0 MB 00:00 2023-10-31T12:03:27.160 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:27.161 INFO:teuthology.orchestra.run.smithi161.stdout:Total 7.9 MB/s | 3.0 MB 00:00 2023-10-31T12:03:27.161 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:27.169 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:27.170 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:27.242 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:27.242 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:27.666 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:27.727 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:27.752 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:27.752 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:03:27.753 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-31T12:03:27.753 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-31T12:03:27.753 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-31T12:03:27.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:28.907 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:03:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:28.907 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:29.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-nbd 2023-10-31T12:03:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:46 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:03:29.857 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:03:29.858 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:29.859 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2023-10-31T12:03:29.859 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:29.859 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:03:29.859 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 168 k 2023-10-31T12:03:29.860 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:29.860 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:03:29.860 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:03:29.861 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2023-10-31T12:03:29.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:29.861 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 168 k 2023-10-31T12:03:29.861 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 532 k 2023-10-31T12:03:29.862 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:03:30.114 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-nbd-18.0.0-6792.ge83bf482.el9.x86_64.rpm 662 kB/s | 168 kB 00:00 2023-10-31T12:03:30.115 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:03:30.115 INFO:teuthology.orchestra.run.smithi161.stdout:Total 658 kB/s | 168 kB 00:00 2023-10-31T12:03:30.115 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:03:30.122 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:03:30.122 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:03:30.178 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:03:30.178 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:03:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:03:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:31.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:03:31.225 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:31.225 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:03:31.225 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:03:31.225 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:31.225 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:03:31.465 DEBUG:teuthology.parallel:result is None 2023-10-31T12:03:31.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=e83bf4821bfb604694b03a67d7390bfb38655506 2023-10-31T12:03:31.615 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-31T12:03:31.636 INFO:teuthology.orchestra.run.smithi161.stdout:18.0.0-6792.ge83bf482.el9 2023-10-31T12:03:31.636 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6792.ge83bf482.el9 2023-10-31T12:03:31.637 INFO:teuthology.task.install:The correct ceph version 18.0.0-6792.ge83bf482 is installed. 2023-10-31T12:03:31.637 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-31T12:03:31.638 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:31.638 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-31T12:03:31.703 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-31T12:03:31.704 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:31.704 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2023-10-31T12:03:31.770 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-31T12:03:31.837 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-31T12:03:31.838 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:31.838 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-31T12:03:31.905 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-31T12:03:31.971 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-31T12:03:31.972 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:31.972 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2023-10-31T12:03:32.038 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-31T12:03:32.105 INFO:teuthology.run_tasks:Running task ceph... 2023-10-31T12:03:32.174 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-31T12:03:32.175 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2023-10-31T12:03:32.208 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-31T12:03:32.208 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-31T12:03:32.274 INFO:tasks.ceph:Creating extra log directories... 2023-10-31T12:03:32.275 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-31T12:03:32.350 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-31T12:03:32.350 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-31T12:03:32.351 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957', 'branch': 'wip-wip-matanb-crimson-do_osd_ops_execute-v3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read}', 'email': None, 'first_in_suite': False, 'job_id': '7441957', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-wip-matanb-crimson-do_osd_ops_execute-v3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '21548fe806cf259deac1421530d5ce720be17997'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1530, 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '21548fe806cf259deac1421530d5ce720be17997', 'targets': {'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIWtOzSIuJbVsTmU7duj9X/WJldXEx3LDhPSRdW79v7wTLOUEUP7milEjJtJazDCQFmwdeXNOi1OaQUkEsW4WxE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-31_11:40:33', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-31T12:03:32.352 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-31T12:03:32.397 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2023-10-31T12:03:32.464 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:32.465 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2023-10-31T12:03:32.520 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-31T12:03:32.520 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2023-10-31T12:03:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-31T12:03:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:03:32.575 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-10-31T12:03:32.575 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:03:32.575 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:03:32.575 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2023-10-31 12:03:27.983348886 +0000 2023-10-31T12:03:32.576 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2023-10-31 11:59:51.015637374 +0000 2023-10-31T12:03:32.576 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2023-10-31 11:59:51.015637374 +0000 2023-10-31T12:03:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2023-10-31 11:59:51.015637374 +0000 2023-10-31T12:03:32.576 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-31T12:03:32.640 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2023-10-31T12:03:32.640 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2023-10-31T12:03:32.640 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000186229 s, 2.7 MB/s 2023-10-31T12:03:32.641 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-31T12:03:32.697 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2023-10-31T12:03:32.753 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-31T12:03:32.753 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:03:32.753 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-31T12:03:32.753 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:03:32.754 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:03:32.754 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2023-10-31 12:03:27.983348886 +0000 2023-10-31T12:03:32.754 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2023-10-31 11:59:51.506627421 +0000 2023-10-31T12:03:32.755 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2023-10-31 11:59:51.506627421 +0000 2023-10-31T12:03:32.755 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2023-10-31 11:59:51.506627421 +0000 2023-10-31T12:03:32.755 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-31T12:03:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2023-10-31T12:03:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2023-10-31T12:03:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000188117 s, 2.7 MB/s 2023-10-31T12:03:32.819 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-31T12:03:32.875 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2023-10-31T12:03:32.929 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-31T12:03:32.930 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:03:32.930 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-31T12:03:32.930 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:03:32.930 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:03:32.930 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2023-10-31 12:03:27.983348886 +0000 2023-10-31T12:03:32.931 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2023-10-31 11:59:51.991617589 +0000 2023-10-31T12:03:32.931 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2023-10-31 11:59:51.991617589 +0000 2023-10-31T12:03:32.931 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2023-10-31 11:59:51.991617589 +0000 2023-10-31T12:03:32.931 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-31T12:03:32.995 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2023-10-31T12:03:32.995 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2023-10-31T12:03:32.995 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000172039 s, 3.0 MB/s 2023-10-31T12:03:32.996 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-31T12:03:33.052 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2023-10-31T12:03:33.106 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-31T12:03:33.106 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:03:33.107 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-10-31T12:03:33.107 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:03:33.107 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:03:33.107 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2023-10-31 12:03:27.984348865 +0000 2023-10-31T12:03:33.108 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2023-10-31 11:59:52.476607758 +0000 2023-10-31T12:03:33.108 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2023-10-31 11:59:52.476607758 +0000 2023-10-31T12:03:33.108 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2023-10-31 11:59:52.476607758 +0000 2023-10-31T12:03:33.108 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-31T12:03:33.171 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2023-10-31T12:03:33.172 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2023-10-31T12:03:33.172 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00018353 s, 2.8 MB/s 2023-10-31T12:03:33.173 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-31T12:03:33.228 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:03:33.228 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-31T12:03:33.229 INFO:tasks.ceph:Generating config... 2023-10-31T12:03:33.230 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-31T12:03:33.230 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-31T12:03:33.231 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-10-31T12:03:33.231 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-31T12:03:33.231 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-31T12:03:33.231 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-10-31T12:03:33.232 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-31T12:03:33.232 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-31T12:03:33.232 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-31T12:03:33.232 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-10-31T12:03:33.233 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-10-31T12:03:33.233 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-10-31T12:03:33.233 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-10-31T12:03:33.233 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-31T12:03:33.234 INFO:tasks.ceph:[osd] err to stderr = True 2023-10-31T12:03:33.234 INFO:tasks.ceph:[osd] log flush on exit = True 2023-10-31T12:03:33.234 INFO:tasks.ceph:[osd] log to file = False 2023-10-31T12:03:33.234 INFO:tasks.ceph:[osd] log to stderr = True 2023-10-31T12:03:33.235 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-31T12:03:33.235 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-31T12:03:33.235 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-31T12:03:33.235 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-31T12:03:33.236 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-31T12:03:33.236 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-31T12:03:33.236 INFO:tasks.ceph:[global] auth client required = none 2023-10-31T12:03:33.236 INFO:tasks.ceph:[global] auth cluster required = none 2023-10-31T12:03:33.237 INFO:tasks.ceph:[global] auth service required = none 2023-10-31T12:03:33.237 INFO:tasks.ceph:[global] auth supported = none 2023-10-31T12:03:33.237 INFO:tasks.ceph:[global] debug asok = 0/0 2023-10-31T12:03:33.237 INFO:tasks.ceph:[global] debug auth = 0/0 2023-10-31T12:03:33.238 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-10-31T12:03:33.238 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-10-31T12:03:33.238 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-10-31T12:03:33.238 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-10-31T12:03:33.239 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-10-31T12:03:33.239 INFO:tasks.ceph:[global] debug client = 0/0 2023-10-31T12:03:33.239 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-10-31T12:03:33.239 INFO:tasks.ceph:[global] debug context = 0/0 2023-10-31T12:03:33.240 INFO:tasks.ceph:[global] debug crush = 0/0 2023-10-31T12:03:33.240 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-10-31T12:03:33.240 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-10-31T12:03:33.240 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-10-31T12:03:33.241 INFO:tasks.ceph:[global] debug filer = 0/0 2023-10-31T12:03:33.241 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-10-31T12:03:33.241 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-10-31T12:03:33.241 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-10-31T12:03:33.242 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-10-31T12:03:33.242 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-10-31T12:03:33.242 INFO:tasks.ceph:[global] debug journal = 0/0 2023-10-31T12:03:33.243 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-10-31T12:03:33.243 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-10-31T12:03:33.243 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-10-31T12:03:33.243 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-10-31T12:03:33.244 INFO:tasks.ceph:[global] debug mds = 0/0 2023-10-31T12:03:33.244 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-10-31T12:03:33.244 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-10-31T12:03:33.244 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-10-31T12:03:33.245 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-10-31T12:03:33.245 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-10-31T12:03:33.245 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-10-31T12:03:33.245 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-10-31T12:03:33.246 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-10-31T12:03:33.246 INFO:tasks.ceph:[global] debug mon = 0/0 2023-10-31T12:03:33.246 INFO:tasks.ceph:[global] debug monc = 0/0 2023-10-31T12:03:33.246 INFO:tasks.ceph:[global] debug ms = 0/0 2023-10-31T12:03:33.247 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-10-31T12:03:33.247 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-10-31T12:03:33.247 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-10-31T12:03:33.248 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-10-31T12:03:33.248 INFO:tasks.ceph:[global] debug osd = 0/0 2023-10-31T12:03:33.248 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-10-31T12:03:33.248 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-10-31T12:03:33.249 INFO:tasks.ceph:[global] debug rados = 0/0 2023-10-31T12:03:33.249 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-10-31T12:03:33.249 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-10-31T12:03:33.249 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-10-31T12:03:33.250 INFO:tasks.ceph:[global] debug refs = 0/0 2023-10-31T12:03:33.250 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-10-31T12:03:33.250 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-10-31T12:03:33.250 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-10-31T12:03:33.251 INFO:tasks.ceph:[global] debug striper = 0/0 2023-10-31T12:03:33.251 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-10-31T12:03:33.251 INFO:tasks.ceph:[global] debug timer = 0/0 2023-10-31T12:03:33.251 INFO:tasks.ceph:[global] debug tp = 0/0 2023-10-31T12:03:33.252 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-31T12:03:33.252 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-31T12:03:33.252 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-31T12:03:33.252 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-31T12:03:33.253 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-31T12:03:33.253 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-31T12:03:33.253 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-31T12:03:33.254 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-31T12:03:33.254 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-31T12:03:33.254 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-10-31T12:03:33.254 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-10-31T12:03:33.255 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-10-31T12:03:33.255 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-31T12:03:33.255 INFO:tasks.ceph:Setting up mon.a... 2023-10-31T12:03:33.256 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-31T12:03:33.324 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.keyring 2023-10-31T12:03:33.327 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-31T12:03:33.377 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-31T12:03:33.403 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.161')] 2023-10-31T12:03:33.404 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.161', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-31T12:03:33.405 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:33.406 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-31T12:03:33.458 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.161 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:33.544 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:33.543+0000 7ff0dfa2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:33.544 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:33.543+0000 7ff0dfa2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:33.545 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:33.543+0000 7ff0dfa2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:33.546 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:33.546 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: generated fsid 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc 2023-10-31T12:03:33.546 INFO:teuthology.orchestra.run.smithi161.stdout:setting min_mon_release = pacific 2023-10-31T12:03:33.547 INFO:teuthology.orchestra.run.smithi161.stdout:epoch 0 2023-10-31T12:03:33.547 INFO:teuthology.orchestra.run.smithi161.stdout:fsid 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc 2023-10-31T12:03:33.547 INFO:teuthology.orchestra.run.smithi161.stdout:last_changed 2023-10-31T12:03:33.544265+0000 2023-10-31T12:03:33.547 INFO:teuthology.orchestra.run.smithi161.stdout:created 2023-10-31T12:03:33.544265+0000 2023-10-31T12:03:33.547 INFO:teuthology.orchestra.run.smithi161.stdout:min_mon_release 16 (pacific) 2023-10-31T12:03:33.548 INFO:teuthology.orchestra.run.smithi161.stdout:election_strategy: 1 2023-10-31T12:03:33.548 INFO:teuthology.orchestra.run.smithi161.stdout:0: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.a 2023-10-31T12:03:33.548 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-31T12:03:33.549 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-31T12:03:33.563 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc... 2023-10-31T12:03:33.564 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-31T12:03:33.653 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2023-10-31T12:03:33.653 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2023-10-31T12:03:33.654 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-31T12:03:33.654 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = none 2023-10-31T12:03:33.654 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.654 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2023-10-31T12:03:33.654 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.655 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2023-10-31T12:03:33.655 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.655 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2023-10-31T12:03:33.655 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2023-10-31T12:03:33.655 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.656 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2023-10-31T12:03:33.656 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2023-10-31T12:03:33.656 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.656 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-31T12:03:33.656 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2023-10-31T12:03:33.656 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.657 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2023-10-31T12:03:33.657 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-31T12:03:33.657 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.657 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2023-10-31T12:03:33.657 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.657 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2023-10-31T12:03:33.658 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2023-10-31T12:03:33.658 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2023-10-31T12:03:33.658 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2023-10-31T12:03:33.658 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2023-10-31T12:03:33.658 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2023-10-31T12:03:33.659 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2023-10-31T12:03:33.659 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-31T12:03:33.659 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-31T12:03:33.659 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2023-10-31T12:03:33.659 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.660 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-31T12:03:33.660 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.660 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2023-10-31T12:03:33.660 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.661 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2023-10-31T12:03:33.661 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.661 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2023-10-31T12:03:33.661 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2023-10-31T12:03:33.661 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2023-10-31T12:03:33.662 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = 172.21.15.161 2023-10-31T12:03:33.662 INFO:teuthology.orchestra.run.smithi161.stdout: auth client required = none 2023-10-31T12:03:33.662 INFO:teuthology.orchestra.run.smithi161.stdout: auth cluster required = none 2023-10-31T12:03:33.662 INFO:teuthology.orchestra.run.smithi161.stdout: auth service required = none 2023-10-31T12:03:33.663 INFO:teuthology.orchestra.run.smithi161.stdout: debug asok = 0/0 2023-10-31T12:03:33.663 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 0/0 2023-10-31T12:03:33.663 INFO:teuthology.orchestra.run.smithi161.stdout: debug bdev = 0/0 2023-10-31T12:03:33.663 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 0/0 2023-10-31T12:03:33.663 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 0/0 2023-10-31T12:03:33.664 INFO:teuthology.orchestra.run.smithi161.stdout: debug buffer = 0/0 2023-10-31T12:03:33.664 INFO:teuthology.orchestra.run.smithi161.stdout: debug civetweb = 0/0 2023-10-31T12:03:33.664 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 0/0 2023-10-31T12:03:33.664 INFO:teuthology.orchestra.run.smithi161.stdout: debug compressor = 0/0 2023-10-31T12:03:33.665 INFO:teuthology.orchestra.run.smithi161.stdout: debug context = 0/0 2023-10-31T12:03:33.665 INFO:teuthology.orchestra.run.smithi161.stdout: debug crush = 0/0 2023-10-31T12:03:33.665 INFO:teuthology.orchestra.run.smithi161.stdout: debug crypto = 0/0 2023-10-31T12:03:33.665 INFO:teuthology.orchestra.run.smithi161.stdout: debug dpdk = 0/0 2023-10-31T12:03:33.665 INFO:teuthology.orchestra.run.smithi161.stdout: debug eventtrace = 0/0 2023-10-31T12:03:33.666 INFO:teuthology.orchestra.run.smithi161.stdout: debug filer = 0/0 2023-10-31T12:03:33.666 INFO:teuthology.orchestra.run.smithi161.stdout: debug filestore = 0/0 2023-10-31T12:03:33.666 INFO:teuthology.orchestra.run.smithi161.stdout: debug finisher = 0/0 2023-10-31T12:03:33.666 INFO:teuthology.orchestra.run.smithi161.stdout: debug fuse = 0/0 2023-10-31T12:03:33.667 INFO:teuthology.orchestra.run.smithi161.stdout: debug heartbeatmap = 0/0 2023-10-31T12:03:33.667 INFO:teuthology.orchestra.run.smithi161.stdout: debug javaclient = 0/0 2023-10-31T12:03:33.667 INFO:teuthology.orchestra.run.smithi161.stdout: debug journal = 0/0 2023-10-31T12:03:33.667 INFO:teuthology.orchestra.run.smithi161.stdout: debug journaler = 0/0 2023-10-31T12:03:33.667 INFO:teuthology.orchestra.run.smithi161.stdout: debug kstore = 0/0 2023-10-31T12:03:33.668 INFO:teuthology.orchestra.run.smithi161.stdout: debug leveldb = 0/0 2023-10-31T12:03:33.668 INFO:teuthology.orchestra.run.smithi161.stdout: debug lockdep = 0/0 2023-10-31T12:03:33.668 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds = 0/0 2023-10-31T12:03:33.668 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds balancer = 0/0 2023-10-31T12:03:33.668 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds locker = 0/0 2023-10-31T12:03:33.669 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds log = 0/0 2023-10-31T12:03:33.669 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds log expire = 0/0 2023-10-31T12:03:33.669 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds migrator = 0/0 2023-10-31T12:03:33.669 INFO:teuthology.orchestra.run.smithi161.stdout: debug memdb = 0/0 2023-10-31T12:03:33.669 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 0/0 2023-10-31T12:03:33.670 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgrc = 0/0 2023-10-31T12:03:33.670 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 0/0 2023-10-31T12:03:33.670 INFO:teuthology.orchestra.run.smithi161.stdout: debug monc = 0/0 2023-10-31T12:03:33.670 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 0/0 2023-10-31T12:03:33.670 INFO:teuthology.orchestra.run.smithi161.stdout: debug objclass = 0/0 2023-10-31T12:03:33.671 INFO:teuthology.orchestra.run.smithi161.stdout: debug objectcacher = 0/0 2023-10-31T12:03:33.671 INFO:teuthology.orchestra.run.smithi161.stdout: debug objecter = 0/0 2023-10-31T12:03:33.671 INFO:teuthology.orchestra.run.smithi161.stdout: debug optracker = 0/0 2023-10-31T12:03:33.671 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 0/0 2023-10-31T12:03:33.671 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 0/0 2023-10-31T12:03:33.672 INFO:teuthology.orchestra.run.smithi161.stdout: debug perfcounter = 0/0 2023-10-31T12:03:33.672 INFO:teuthology.orchestra.run.smithi161.stdout: debug rados = 0/0 2023-10-31T12:03:33.672 INFO:teuthology.orchestra.run.smithi161.stdout: debug rbd = 0/0 2023-10-31T12:03:33.672 INFO:teuthology.orchestra.run.smithi161.stdout: debug rbd mirror = 0/0 2023-10-31T12:03:33.672 INFO:teuthology.orchestra.run.smithi161.stdout: debug rbd replay = 0/0 2023-10-31T12:03:33.673 INFO:teuthology.orchestra.run.smithi161.stdout: debug refs = 0/0 2023-10-31T12:03:33.673 INFO:teuthology.orchestra.run.smithi161.stdout: debug rgw = 0/0 2023-10-31T12:03:33.673 INFO:teuthology.orchestra.run.smithi161.stdout: debug rgw sync = 0/0 2023-10-31T12:03:33.673 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 0/0 2023-10-31T12:03:33.673 INFO:teuthology.orchestra.run.smithi161.stdout: debug striper = 0/0 2023-10-31T12:03:33.674 INFO:teuthology.orchestra.run.smithi161.stdout: debug throttle = 0/0 2023-10-31T12:03:33.674 INFO:teuthology.orchestra.run.smithi161.stdout: debug timer = 0/0 2023-10-31T12:03:33.674 INFO:teuthology.orchestra.run.smithi161.stdout: debug tp = 0/0 2023-10-31T12:03:33.674 INFO:teuthology.orchestra.run.smithi161.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-31T12:03:33.674 INFO:teuthology.orchestra.run.smithi161.stdout: ms client mode = crc 2023-10-31T12:03:33.675 INFO:teuthology.orchestra.run.smithi161.stdout: ms cluster mode = crc 2023-10-31T12:03:33.675 INFO:teuthology.orchestra.run.smithi161.stdout: ms mon client mode = crc 2023-10-31T12:03:33.675 INFO:teuthology.orchestra.run.smithi161.stdout: ms mon cluster mode = crc 2023-10-31T12:03:33.675 INFO:teuthology.orchestra.run.smithi161.stdout: ms mon service mode = crc 2023-10-31T12:03:33.675 INFO:teuthology.orchestra.run.smithi161.stdout: ms service mode = crc 2023-10-31T12:03:33.675 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc 2023-10-31T12:03:33.676 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.676 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2023-10-31T12:03:33.676 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2023-10-31T12:03:33.676 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.677 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2023-10-31T12:03:33.677 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2023-10-31T12:03:33.677 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2023-10-31T12:03:33.677 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.677 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2023-10-31T12:03:33.677 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2023-10-31T12:03:33.678 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.678 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2023-10-31T12:03:33.678 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2023-10-31T12:03:33.678 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2023-10-31T12:03:33.679 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.679 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2023-10-31T12:03:33.679 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2023-10-31T12:03:33.679 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.679 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2023-10-31T12:03:33.680 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.680 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2023-10-31T12:03:33.680 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.680 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2023-10-31T12:03:33.681 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.681 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2023-10-31T12:03:33.681 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2023-10-31T12:03:33.681 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.681 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2023-10-31T12:03:33.682 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2023-10-31T12:03:33.682 INFO:teuthology.orchestra.run.smithi161.stdout: debug monc = 20 2023-10-31T12:03:33.682 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore block size = 96636764160 2023-10-31T12:03:33.682 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore compression algorithm = snappy 2023-10-31T12:03:33.682 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore compression mode = aggressive 2023-10-31T12:03:33.683 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore fsck on mount = True 2023-10-31T12:03:33.683 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore rocksdb cf = False 2023-10-31T12:03:33.683 INFO:teuthology.orchestra.run.smithi161.stdout: crimson seastar smp = 3 2023-10-31T12:03:33.683 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 20 2023-10-31T12:03:33.683 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 20 2023-10-31T12:03:33.684 INFO:teuthology.orchestra.run.smithi161.stdout: debug filestore = 0/0 2023-10-31T12:03:33.684 INFO:teuthology.orchestra.run.smithi161.stdout: debug journal = 0/0 2023-10-31T12:03:33.684 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 0/0 2023-10-31T12:03:33.684 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 0/0 2023-10-31T12:03:33.684 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 10 2023-10-31T12:03:33.685 INFO:teuthology.orchestra.run.smithi161.stdout: err to stderr = True 2023-10-31T12:03:33.685 INFO:teuthology.orchestra.run.smithi161.stdout: log flush on exit = True 2023-10-31T12:03:33.685 INFO:teuthology.orchestra.run.smithi161.stdout: log to file = False 2023-10-31T12:03:33.685 INFO:teuthology.orchestra.run.smithi161.stdout: log to stderr = True 2023-10-31T12:03:33.685 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd backfillfull_ratio = 0.85 2023-10-31T12:03:33.686 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd full ratio = 0.9 2023-10-31T12:03:33.686 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd nearfull ratio = 0.8 2023-10-31T12:03:33.686 INFO:teuthology.orchestra.run.smithi161.stdout: osd failsafe full ratio = 0.95 2023-10-31T12:03:33.686 INFO:teuthology.orchestra.run.smithi161.stdout: osd objectstore = bluestore 2023-10-31T12:03:33.686 INFO:teuthology.orchestra.run.smithi161.stdout: osd shutdown pgref assert = True 2023-10-31T12:03:33.687 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.687 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2023-10-31T12:03:33.687 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2023-10-31T12:03:33.687 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2023-10-31T12:03:33.687 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2023-10-31T12:03:33.688 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2023-10-31T12:03:33.688 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2023-10-31T12:03:33.688 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2023-10-31T12:03:33.688 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2023-10-31T12:03:33.688 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.689 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2023-10-31T12:03:33.689 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 0/0 2023-10-31T12:03:33.689 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 0/0 2023-10-31T12:03:33.689 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 0/0 2023-10-31T12:03:33.690 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2023-10-31T12:03:33.690 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2023-10-31T12:03:33.690 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2023-10-31T12:03:33.690 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2023-10-31T12:03:33.690 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2023-10-31T12:03:33.691 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2023-10-31T12:03:33.691 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2023-10-31T12:03:33.691 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.691 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-31T12:03:33.692 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2023-10-31T12:03:33.692 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2023-10-31T12:03:33.692 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.692 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2023-10-31T12:03:33.692 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-31T12:03:33.692 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-31T12:03:33.693 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.693 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2023-10-31T12:03:33.693 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2023-10-31T12:03:33.693 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.694 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2023-10-31T12:03:33.694 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default crimson = True 2023-10-31T12:03:33.694 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:33.694 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2023-10-31T12:03:33.694 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2023-10-31T12:03:33.694 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2023-10-31T12:03:33.695 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2023-10-31T12:03:33.695 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-31T12:03:33.695 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-31T12:03:33.695 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2023-10-31T12:03:33.696 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-31T12:03:33.697 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-31T12:03:33.759 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-31T12:03:33.760 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:33.760 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-31T12:03:33.778 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:33.778 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-31T12:03:33.833 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:03:33.833 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:33.833 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-31T12:03:33.834 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-31T12:03:33.912 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:33.912 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:33.966 INFO:tasks.ceph:Setting up mon nodes... 2023-10-31T12:03:33.967 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-31T12:03:33.967 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-31T12:03:34.067 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-31T12:03:34.070 INFO:tasks.ceph:Setting up mds nodes... 2023-10-31T12:03:34.070 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-31T12:03:34.070 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-31T12:03:34.118 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-31T12:03:34.134 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-31T12:03:34.135 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-31T12:03:34.135 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-31T12:03:34.161 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:03:34.162 INFO:tasks.ceph:role: osd.0 2023-10-31T12:03:34.162 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:03:34.162 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-31T12:03:34.236 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T12:03:34.236 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T12:03:34.236 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T12:03:34.236 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T12:03:34.237 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T12:03:34.237 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2023-10-31T12:03:34.237 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T12:03:34.237 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T12:03:34.237 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T12:03:34.238 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T12:03:34.723 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2023-10-31T12:03:34.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2023-10-31T12:03:34.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-31T12:03:34.864 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-31T12:03:34.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-31T12:03:34.965 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:03:34.966 INFO:tasks.ceph:role: osd.1 2023-10-31T12:03:34.966 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:03:34.967 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-31T12:03:35.040 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T12:03:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T12:03:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T12:03:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T12:03:35.041 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T12:03:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2023-10-31T12:03:35.042 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T12:03:35.042 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T12:03:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T12:03:35.042 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T12:03:35.502 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2023-10-31T12:03:35.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2023-10-31T12:03:35.504 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-31T12:03:35.595 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-31T12:03:35.626 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-31T12:03:35.693 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:03:35.694 INFO:tasks.ceph:role: osd.2 2023-10-31T12:03:35.694 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:03:35.694 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-31T12:03:35.768 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T12:03:35.768 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T12:03:35.768 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T12:03:35.769 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T12:03:35.769 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T12:03:35.769 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2023-10-31T12:03:35.769 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T12:03:35.770 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T12:03:35.770 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T12:03:35.770 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T12:03:36.259 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2023-10-31T12:03:36.261 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2023-10-31T12:03:36.262 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-31T12:03:36.393 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-31T12:03:36.428 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:36.942 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:03:36.943 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:36,942 seastar - Reactor backend: linux-aio 2023-10-31T12:03:36.962 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:36,962 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:03:36.963 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:36,962 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:03:36.963 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:36,962 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:03:36.963 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:36,962 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:03:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:==80211==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:03:36.971 INFO:teuthology.orchestra.run.smithi161.stderr:==80211==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe1021f000; bottom 0x7f030b752000; size: 0x00fb04acd000 (1078115225600) 2023-10-31T12:03:36.971 INFO:teuthology.orchestra.run.smithi161.stderr:False positive error reports may follow 2023-10-31T12:03:36.971 INFO:teuthology.orchestra.run.smithi161.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:03:37.484 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:37.484 INFO:teuthology.orchestra.run.smithi161.stderr:Suppressions used: 2023-10-31T12:03:37.484 INFO:teuthology.orchestra.run.smithi161.stderr: count bytes template 2023-10-31T12:03:37.484 INFO:teuthology.orchestra.run.smithi161.stderr: 1 8 MallocExtension::Register 2023-10-31T12:03:37.485 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:37.485 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:03:37.537 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:03:37.538 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:37,538 seastar - Reactor backend: linux-aio 2023-10-31T12:03:37.566 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:37,565 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:03:37.566 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:37,565 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:03:37.566 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:37,565 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:03:37.573 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:37,573 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:03:37.576 INFO:teuthology.orchestra.run.smithi161.stderr:==80209==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:==80209==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe1021f000; bottom 0x7f030953b000; size: 0x00fb06ce4000 (1078150971392) 2023-10-31T12:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:False positive error reports may follow 2023-10-31T12:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:03:41.592 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:41.592 INFO:teuthology.orchestra.run.smithi161.stderr:Suppressions used: 2023-10-31T12:03:41.592 INFO:teuthology.orchestra.run.smithi161.stderr: count bytes template 2023-10-31T12:03:41.593 INFO:teuthology.orchestra.run.smithi161.stderr: 1 8 MallocExtension::Register 2023-10-31T12:03:41.593 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:41.593 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:03:41.593 INFO:teuthology.orchestra.run.smithi161.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc 2023-10-31T12:03:41.629 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-31T12:03:41.656 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:42.162 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:03:42.163 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,162 seastar - Reactor backend: linux-aio 2023-10-31T12:03:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,178 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:03:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,178 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:03:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,178 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:03:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,178 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:03:42.181 INFO:teuthology.orchestra.run.smithi161.stderr:==80393==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:03:42.187 INFO:teuthology.orchestra.run.smithi161.stderr:==80393==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff08aeb000; bottom 0x7f3b2dfc5000; size: 0x00c3dab26000 (841187745792) 2023-10-31T12:03:42.187 INFO:teuthology.orchestra.run.smithi161.stderr:False positive error reports may follow 2023-10-31T12:03:42.187 INFO:teuthology.orchestra.run.smithi161.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:03:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:42.693 INFO:teuthology.orchestra.run.smithi161.stderr:Suppressions used: 2023-10-31T12:03:42.693 INFO:teuthology.orchestra.run.smithi161.stderr: count bytes template 2023-10-31T12:03:42.693 INFO:teuthology.orchestra.run.smithi161.stderr: 1 8 MallocExtension::Register 2023-10-31T12:03:42.693 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:42.694 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:03:42.744 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:03:42.745 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,745 seastar - Reactor backend: linux-aio 2023-10-31T12:03:42.763 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,762 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:03:42.763 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,762 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:03:42.763 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,762 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:03:42.770 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:42,770 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:03:42.773 INFO:teuthology.orchestra.run.smithi161.stderr:==80391==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:03:42.779 INFO:teuthology.orchestra.run.smithi161.stderr:==80391==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff08aeb000; bottom 0x7f3b2bdae000; size: 0x00c3dcd3d000 (841223491584) 2023-10-31T12:03:42.779 INFO:teuthology.orchestra.run.smithi161.stderr:False positive error reports may follow 2023-10-31T12:03:42.779 INFO:teuthology.orchestra.run.smithi161.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:03:46.773 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:46.773 INFO:teuthology.orchestra.run.smithi161.stderr:Suppressions used: 2023-10-31T12:03:46.773 INFO:teuthology.orchestra.run.smithi161.stderr: count bytes template 2023-10-31T12:03:46.773 INFO:teuthology.orchestra.run.smithi161.stderr: 1 8 MallocExtension::Register 2023-10-31T12:03:46.774 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:46.774 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:03:46.774 INFO:teuthology.orchestra.run.smithi161.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc 2023-10-31T12:03:46.823 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-31T12:03:46.851 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:47.354 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:03:47.355 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,354 seastar - Reactor backend: linux-aio 2023-10-31T12:03:47.370 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,370 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:03:47.371 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,370 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:03:47.371 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,370 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:03:47.371 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,370 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:03:47.372 INFO:teuthology.orchestra.run.smithi161.stderr:==80576==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:03:47.378 INFO:teuthology.orchestra.run.smithi161.stderr:==80576==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc9275c000; bottom 0x7f6271a6a000; size: 0x009a20cf2000 (661975408640) 2023-10-31T12:03:47.379 INFO:teuthology.orchestra.run.smithi161.stderr:False positive error reports may follow 2023-10-31T12:03:47.379 INFO:teuthology.orchestra.run.smithi161.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:03:47.405 INFO:teuthology.orchestra.run.smithi161.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0x22bbc912 0x22c9eadc 0x22ca4d45 0x228ffe94 0x22972ed3 0x24973b31 0x24979259 0x2497b7f4 0x2497c9cd 0x224fd1f1 0x22538a7e 0x22560df5 0x2c5dea65 0x2c67c872 0x2c8a0672 0x2c8a24bd 0x2c2faad6 0x2c2fc9d7 0x1ed6c220 0x1ed7638d 0x1ebf325d 0x3feaf 0x3ff5f 0x1e6ada04 2023-10-31T12:03:47.405 INFO:teuthology.orchestra.run.smithi161.stderr:kernel callstack: 2023-10-31T12:03:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:Suppressions used: 2023-10-31T12:03:47.893 INFO:teuthology.orchestra.run.smithi161.stderr: count bytes template 2023-10-31T12:03:47.893 INFO:teuthology.orchestra.run.smithi161.stderr: 1 8 MallocExtension::Register 2023-10-31T12:03:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:47.894 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:03:47.946 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:03:47.947 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,947 seastar - Reactor backend: linux-aio 2023-10-31T12:03:47.974 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,974 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:03:47.975 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,974 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:03:47.975 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,974 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:03:47.982 INFO:teuthology.orchestra.run.smithi161.stderr:INFO 2023-10-31 12:03:47,982 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:03:47.985 INFO:teuthology.orchestra.run.smithi161.stderr:==80574==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:03:47.991 INFO:teuthology.orchestra.run.smithi161.stderr:==80574==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc9275c000; bottom 0x7f626f853000; size: 0x009a22f09000 (662011154432) 2023-10-31T12:03:47.991 INFO:teuthology.orchestra.run.smithi161.stderr:False positive error reports may follow 2023-10-31T12:03:47.991 INFO:teuthology.orchestra.run.smithi161.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:03:51.984 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:51.984 INFO:teuthology.orchestra.run.smithi161.stderr:Suppressions used: 2023-10-31T12:03:51.984 INFO:teuthology.orchestra.run.smithi161.stderr: count bytes template 2023-10-31T12:03:51.984 INFO:teuthology.orchestra.run.smithi161.stderr: 1 8 MallocExtension::Register 2023-10-31T12:03:51.985 INFO:teuthology.orchestra.run.smithi161.stderr:----------------------------------------------------- 2023-10-31T12:03:51.985 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:03:51.985 INFO:teuthology.orchestra.run.smithi161.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 6ecd1e33-603a-4a88-a39e-6ba6a078a0fc 2023-10-31T12:03:52.019 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-31T12:03:52.046 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-31T12:03:52.046 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:52.047 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-31T12:03:52.113 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:52.113 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-31T12:03:52.180 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:52.180 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-31T12:03:52.246 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:52.246 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-31T12:03:52.312 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:52.313 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-31T12:03:52.367 INFO:tasks.ceph:Adding keys to all mons... 2023-10-31T12:03:52.367 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-31T12:03:52.431 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2023-10-31T12:03:52.432 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQAW7UBl2DL8AxAArNmMnLRTZ66B0sSXI//0Yg== 2023-10-31T12:03:52.432 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2023-10-31T12:03:52.432 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQAZ7UBlQmphJRAAYeBt9i241a58nfw0nAqEnA== 2023-10-31T12:03:52.432 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2023-10-31T12:03:52.433 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQAe7UBlhBFIMRAApDd3bBlTjfYa5z7LLpyfgg== 2023-10-31T12:03:52.433 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2023-10-31T12:03:52.433 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQAk7UBlPjcmAhAA8dmRLdvmOEC8XKpdA8JOjg== 2023-10-31T12:03:52.433 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2023-10-31T12:03:52.433 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQAW7UBlI8UKBxAARSN3Sps2HKpTMHWEU/XDpg== 2023-10-31T12:03:52.434 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-31T12:03:52.532 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T12:03:52.589 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T12:03:52.647 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T12:03:52.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-31T12:03:52.766 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-31T12:03:52.766 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-31T12:03:52.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-31T12:03:52.896 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:52.894+0000 7f324400d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:52.897 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:52.894+0000 7f324400d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:52.911 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-31T12:03:52.938 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:03:52.993 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-31T12:03:52.994 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-31T12:03:52.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-31T12:03:53.035 INFO:tasks.ceph.mon.a:Started 2023-10-31T12:03:53.035 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-31T12:03:53.036 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-31T12:03:53.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-31T12:03:53.038 INFO:tasks.ceph.mgr.x:Started 2023-10-31T12:03:53.039 INFO:tasks.ceph:Setting crush tunables to default 2023-10-31T12:03:53.039 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd crush tunables default 2023-10-31T12:03:53.116 INFO:tasks.ceph.mon.a.smithi161.stderr:2023-10-31T12:03:53.114+0000 7fd9aba17980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.117 INFO:tasks.ceph.mon.a.smithi161.stderr:2023-10-31T12:03:53.114+0000 7fd9aba17980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.117 INFO:tasks.ceph.mon.a.smithi161.stderr:2023-10-31T12:03:53.114+0000 7fd9aba17980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.117 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:53.114+0000 7fa26b084f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.153 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:53.146+0000 7f2ed0d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.190 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:53.189+0000 7f2ed0d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.258 INFO:teuthology.orchestra.run.smithi161.stderr:adjusted tunables profile to default 2023-10-31T12:03:53.270 INFO:tasks.ceph:check_enable_crimson: True 2023-10-31T12:03:53.270 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-31T12:03:53.271 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-31T12:03:53.319 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:53.317+0000 7fa26b084f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.320 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:53.317+0000 7fa26b084f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.377 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:53.370+0000 7ff60915e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.378 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:53.376+0000 7ff60915e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:53.458 INFO:tasks.ceph.mon.a.smithi161.stderr:2023-10-31T12:03:53.456+0000 7fd9a39f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-31T12:03:53.458 INFO:tasks.ceph.mon.a.smithi161.stderr:Please be aware that this feature is experimental, untested, 2023-10-31T12:03:53.459 INFO:tasks.ceph.mon.a.smithi161.stderr:unsupported, and may result in data corruption, data loss, 2023-10-31T12:03:53.459 INFO:tasks.ceph.mon.a.smithi161.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-31T12:03:53.459 INFO:tasks.ceph.mon.a.smithi161.stderr:feature with important data. 2023-10-31T12:03:53.459 INFO:tasks.ceph.mon.a.smithi161.stderr: 2023-10-31T12:03:53.784 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:53.782+0000 7fa26b084f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-31T12:03:53.835 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:53.833+0000 7fa26b084f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-31T12:03:54.271 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-31T12:03:54.271 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:54.272 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-31T12:03:54.298 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:54.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-31T12:03:54.365 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:03:54.366 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-31T12:03:54.393 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.391+0000 7fa26b084f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-31T12:03:54.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 8be3bcb9-04c2-459d-a1e1-4d9f04fd5c17 0 2023-10-31T12:03:54.447 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.445+0000 7fa26b084f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-31T12:03:54.494 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.492+0000 7fa26b084f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-31T12:03:54.551 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.549+0000 7fa26b084f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-31T12:03:54.553 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:54.546+0000 7f699591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:54.554 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:54.553+0000 7f699591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:54.607 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.605+0000 7fa26b084f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-31T12:03:54.646 INFO:teuthology.orchestra.run.smithi161.stdout:0 2023-10-31T12:03:54.657 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new aec5002d-01dc-49fb-a3db-19a4b1344b47 1 2023-10-31T12:03:54.712 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.710+0000 7fa26b084f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-31T12:03:54.764 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:54.757+0000 7fc18a75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:54.766 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:54.764+0000 7fc18a75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:54.787 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.785+0000 7fa26b084f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-31T12:03:54.834 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.832+0000 7fa26b084f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-31T12:03:54.842 INFO:teuthology.orchestra.run.smithi161.stdout:1 2023-10-31T12:03:54.853 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new b72fca58-2695-4e53-9544-d7ab92ef53e0 2 2023-10-31T12:03:54.889 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.887+0000 7fa26b084f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-31T12:03:54.938 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:54.936+0000 7fa26b084f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-31T12:03:54.958 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:54.951+0000 7f365d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:54.960 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:54.958+0000 7f365d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:55.035 INFO:teuthology.orchestra.run.smithi161.stdout:2 2023-10-31T12:03:55.047 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-31T12:03:55.118 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-31T12:03:55.119 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-31T12:03:55.160 INFO:tasks.ceph.osd.0:Started 2023-10-31T12:03:55.160 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-31T12:03:55.191 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-31T12:03:55.192 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-31T12:03:55.234 INFO:tasks.ceph.osd.1:Started 2023-10-31T12:03:55.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-31T12:03:55.268 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-31T12:03:55.268 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-31T12:03:55.311 INFO:tasks.ceph.osd.2:Started 2023-10-31T12:03:55.312 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T12:03:55.354 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:55.352+0000 7fa26b084f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-31T12:03:55.509 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:55.500+0000 7f53aff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:03:55.511+0000 7f53aff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:55.609 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:55.607+0000 7fa26b084f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-31T12:03:55.646 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:03:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"fsid":"6ecd1e33-603a-4a88-a39e-6ba6a078a0fc","created":"2023-10-31T12:03:53.175864+0000","modified":"2023-10-31T12:03:55.034490+0000","last_up_change":"0.000000","last_in_change":"2023-10-31T12:03:55.034490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"8be3bcb9-04c2-459d-a1e1-4d9f04fd5c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aec5002d-01dc-49fb-a3db-19a4b1344b47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b72fca58-2695-4e53-9544-d7ab92ef53e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:03:55.670 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-31T12:03:55.671 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-31T12:03:55.788 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:55.786+0000 7fa26b084f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-31T12:03:55.865 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:55.863+0000 7fa26b084f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-31T12:03:55.973 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-31T12:03:56.038 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:56.036+0000 7fa26b084f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-31T12:03:56.071 INFO:teuthology.misc.health.smithi161.stderr:2023-10-31T12:03:56.065+0000 7fe924c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:56.073 INFO:teuthology.misc.health.smithi161.stderr:2023-10-31T12:03:56.071+0000 7fe924c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:03:56.138 INFO:teuthology.misc.health.smithi161.stdout: 2023-10-31T12:03:56.138 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":6,"fsid":"6ecd1e33-603a-4a88-a39e-6ba6a078a0fc","created":"2023-10-31T12:03:53.175864+0000","modified":"2023-10-31T12:03:55.034490+0000","last_up_change":"0.000000","last_in_change":"2023-10-31T12:03:55.034490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"8be3bcb9-04c2-459d-a1e1-4d9f04fd5c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aec5002d-01dc-49fb-a3db-19a4b1344b47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b72fca58-2695-4e53-9544-d7ab92ef53e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:03:56.148 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-31T12:03:56.200 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:56.198+0000 7fa26b084f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-31T12:03:56.436 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:56.434+0000 7fa26b084f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-31T12:03:56.504 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:56.501+0000 7fa26b084f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-31T12:03:56.742 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:56.740+0000 7fa26b084f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-31T12:03:58.037 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:58.035+0000 7fa26b084f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-31T12:03:58.436 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:58.434+0000 7fa26b084f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-31T12:03:58.492 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:58.490+0000 7fa26b084f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-31T12:03:58.598 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:58.596+0000 7fa26b084f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-31T12:03:59.618 INFO:tasks.ceph.mgr.x.smithi161.stderr:2023-10-31T12:03:59.616+0000 7fa20b5f0640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-31T12:04:02.450 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-31T12:04:02.557 INFO:teuthology.misc.health.smithi161.stderr:2023-10-31T12:04:02.552+0000 7fa608f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:02.559 INFO:teuthology.misc.health.smithi161.stderr:2023-10-31T12:04:02.557+0000 7fa608f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:02.789 INFO:teuthology.misc.health.smithi161.stdout: 2023-10-31T12:04:02.790 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":8,"fsid":"6ecd1e33-603a-4a88-a39e-6ba6a078a0fc","created":"2023-10-31T12:03:53.175864+0000","modified":"2023-10-31T12:04:01.618563+0000","last_up_change":"2023-10-31T12:04:01.618563+0000","last_in_change":"2023-10-31T12:03:55.034490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"8be3bcb9-04c2-459d-a1e1-4d9f04fd5c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2152161113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6803","nonce":2152161113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2152161113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":2152161113}]},"public_addr":"172.21.15.161:6802/2152161113","cluster_addr":"172.21.15.161:6803/2152161113","heartbeat_back_addr":"172.21.15.161:6808/2152161113","heartbeat_front_addr":"172.21.15.161:6809/2152161113","state":["exists","up"]},{"osd":1,"uuid":"aec5002d-01dc-49fb-a3db-19a4b1344b47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6805","nonce":4187984997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4187984997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6811","nonce":4187984997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":4187984997}]},"public_addr":"172.21.15.161:6805/4187984997","cluster_addr":"172.21.15.161:6804/4187984997","heartbeat_back_addr":"172.21.15.161:6811/4187984997","heartbeat_front_addr":"172.21.15.161:6810/4187984997","state":["exists","up"]},{"osd":2,"uuid":"b72fca58-2695-4e53-9544-d7ab92ef53e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":677713902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":677713902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":677713902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":677713902}]},"public_addr":"172.21.15.161:6807/677713902","cluster_addr":"172.21.15.161:6806/677713902","heartbeat_back_addr":"172.21.15.161:6813/677713902","heartbeat_front_addr":"172.21.15.161:6812/677713902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:04:02.801 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-31T12:04:02.801 INFO:tasks.ceph:Creating RBD pool 2023-10-31T12:04:02.802 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-31T12:04:02.907 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:02.899+0000 7f4edc664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:02.909 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:02.907+0000 7f4edc664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:03.634 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'rbd' created 2023-10-31T12:04:03.654 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-31T12:04:03.769 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:03.759+0000 7fb6bd02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:03.771 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:03.769+0000 7fb6bd02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:04.638 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-31T12:04:04.653 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-31T12:04:04.654 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-31T12:04:04.654 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-31T12:04:04.654 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-31T12:04:04.655 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-31T12:04:04.787 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:04.782+0000 7f3da9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:04.789 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:04.787+0000 7f3da9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.055 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:05.072 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2144358067},{"type":"v1","addr":"172.21.15.161:6801","nonce":2144358067}]},"active_addr":"172.21.15.161:6801/2144358067","active_change":"2023-10-31T12:03:58.610363+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3180354660}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":652754787}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":1278542098}]}]} 2023-10-31T12:04:05.075 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-31T12:04:05.075 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-31T12:04:05.075 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T12:04:05.196 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.192+0000 7f81d662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.198 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.197+0000 7f81d662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:05.430 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":10,"fsid":"6ecd1e33-603a-4a88-a39e-6ba6a078a0fc","created":"2023-10-31T12:03:53.175864+0000","modified":"2023-10-31T12:04:04.634637+0000","last_up_change":"2023-10-31T12:04:01.618563+0000","last_in_change":"2023-10-31T12:03:55.034490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-31T12:04:02.654026+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-31T12:04:03.100878+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8be3bcb9-04c2-459d-a1e1-4d9f04fd5c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2152161113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6803","nonce":2152161113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2152161113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":2152161113}]},"public_addr":"172.21.15.161:6802/2152161113","cluster_addr":"172.21.15.161:6803/2152161113","heartbeat_back_addr":"172.21.15.161:6808/2152161113","heartbeat_front_addr":"172.21.15.161:6809/2152161113","state":["exists","up"]},{"osd":1,"uuid":"aec5002d-01dc-49fb-a3db-19a4b1344b47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6805","nonce":4187984997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4187984997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6811","nonce":4187984997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":4187984997}]},"public_addr":"172.21.15.161:6805/4187984997","cluster_addr":"172.21.15.161:6804/4187984997","heartbeat_back_addr":"172.21.15.161:6811/4187984997","heartbeat_front_addr":"172.21.15.161:6810/4187984997","state":["exists","up"]},{"osd":2,"uuid":"b72fca58-2695-4e53-9544-d7ab92ef53e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":677713902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":677713902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":677713902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":677713902}]},"public_addr":"172.21.15.161:6807/677713902","cluster_addr":"172.21.15.161:6806/677713902","heartbeat_back_addr":"172.21.15.161:6813/677713902","heartbeat_front_addr":"172.21.15.161:6812/677713902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:04:05.443 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-31T12:04:05.444 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T12:04:05.562 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.558+0000 7f5786711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.565 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.564+0000 7f5786711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.756 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:05.756 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":11,"fsid":"6ecd1e33-603a-4a88-a39e-6ba6a078a0fc","created":"2023-10-31T12:03:53.175864+0000","modified":"2023-10-31T12:04:05.637648+0000","last_up_change":"2023-10-31T12:04:01.618563+0000","last_in_change":"2023-10-31T12:03:55.034490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-31T12:04:02.654026+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-31T12:04:03.100878+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8be3bcb9-04c2-459d-a1e1-4d9f04fd5c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2152161113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6803","nonce":2152161113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2152161113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":2152161113}]},"public_addr":"172.21.15.161:6802/2152161113","cluster_addr":"172.21.15.161:6803/2152161113","heartbeat_back_addr":"172.21.15.161:6808/2152161113","heartbeat_front_addr":"172.21.15.161:6809/2152161113","state":["exists","up"]},{"osd":1,"uuid":"aec5002d-01dc-49fb-a3db-19a4b1344b47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6805","nonce":4187984997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4187984997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6811","nonce":4187984997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":4187984997}]},"public_addr":"172.21.15.161:6805/4187984997","cluster_addr":"172.21.15.161:6804/4187984997","heartbeat_back_addr":"172.21.15.161:6811/4187984997","heartbeat_front_addr":"172.21.15.161:6810/4187984997","state":["exists","up"]},{"osd":2,"uuid":"b72fca58-2695-4e53-9544-d7ab92ef53e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":677713902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":677713902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":677713902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":677713902}]},"public_addr":"172.21.15.161:6807/677713902","cluster_addr":"172.21.15.161:6806/677713902","heartbeat_back_addr":"172.21.15.161:6813/677713902","heartbeat_front_addr":"172.21.15.161:6812/677713902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:04:05.769 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-31T12:04:05.769 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-31T12:04:05.770 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-31T12:04:05.886 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.881+0000 7f489899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.887 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.881+0000 7f91bc4d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.887 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.881+0000 7ff093224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.889 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.887+0000 7f489899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.889 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.887+0000 7ff093224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.889 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:05.887+0000 7f91bc4d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:05.918 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2023-10-31T12:04:05.918 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-31T12:04:05.919 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2023-10-31T12:04:05.920 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-31T12:04:05.921 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2023-10-31T12:04:05.921 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-31T12:04:06.071 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.066+0000 7f0115b55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.071 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.066+0000 7fa0f002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.072 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.066+0000 7fb27ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.073 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.072+0000 7fa0f002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.074 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.072+0000 7fb27ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.074 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.072+0000 7f0115b55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.255 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2023-10-31T12:04:06.261 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2023-10-31T12:04:06.265 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2023-10-31T12:04:06.268 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-10-31T12:04:06.268 DEBUG:teuthology.parallel:result is None 2023-10-31T12:04:06.275 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-10-31T12:04:06.276 DEBUG:teuthology.parallel:result is None 2023-10-31T12:04:06.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-10-31T12:04:06.278 DEBUG:teuthology.parallel:result is None 2023-10-31T12:04:06.278 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-31T12:04:06.279 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-31T12:04:06.423 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.419+0000 7f4e186ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.425 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.424+0000 7f4e186ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:06.597 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2023-10-31T12:04:06.608 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2023-10-31T12:04:05.905878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80740,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034780,"statfs":{"total":289910292480,"available":289827614720,"internally_reserved":0,"allocated":262144,"data_stored":1221192,"data_compressed":59862,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.651993+0000","last_change":"2023-10-31T12:04:04.717456+0000","last_active":"2023-10-31T12:04:05.651993+0000","last_peered":"2023-10-31T12:04:05.651993+0000","last_clean":"2023-10-31T12:04:05.651993+0000","last_became_active":"2023-10-31T12:04:04.715686+0000","last_became_peered":"2023-10-31T12:04:04.715686+0000","last_unstale":"2023-10-31T12:04:05.651993+0000","last_undegraded":"2023-10-31T12:04:05.651993+0000","last_fullsized":"2023-10-31T12:04:05.651993+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.653443+0000","last_change":"2023-10-31T12:04:04.789622+0000","last_active":"2023-10-31T12:04:05.653443+0000","last_peered":"2023-10-31T12:04:05.653443+0000","last_clean":"2023-10-31T12:04:05.653443+0000","last_became_active":"2023-10-31T12:04:04.787675+0000","last_became_peered":"2023-10-31T12:04:04.787675+0000","last_unstale":"2023-10-31T12:04:05.653443+0000","last_undegraded":"2023-10-31T12:04:05.653443+0000","last_fullsized":"2023-10-31T12:04:05.653443+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.652066+0000","last_change":"2023-10-31T12:04:04.736124+0000","last_active":"2023-10-31T12:04:05.652066+0000","last_peered":"2023-10-31T12:04:05.652066+0000","last_clean":"2023-10-31T12:04:05.652066+0000","last_became_active":"2023-10-31T12:04:04.727697+0000","last_became_peered":"2023-10-31T12:04:04.727697+0000","last_unstale":"2023-10-31T12:04:05.652066+0000","last_undegraded":"2023-10-31T12:04:05.652066+0000","last_fullsized":"2023-10-31T12:04:05.652066+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.653757+0000","last_change":"2023-10-31T12:04:04.772694+0000","last_active":"2023-10-31T12:04:05.653757+0000","last_peered":"2023-10-31T12:04:05.653757+0000","last_clean":"2023-10-31T12:04:05.653757+0000","last_became_active":"2023-10-31T12:04:04.768360+0000","last_became_peered":"2023-10-31T12:04:04.768360+0000","last_unstale":"2023-10-31T12:04:05.653757+0000","last_undegraded":"2023-10-31T12:04:05.653757+0000","last_fullsized":"2023-10-31T12:04:05.653757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-31T12:04:04.746534+0000","last_change":"2023-10-31T12:04:04.746754+0000","last_active":"2023-10-31T12:04:04.746534+0000","last_peered":"2023-10-31T12:04:04.746534+0000","last_clean":"2023-10-31T12:04:03.624440+0000","last_became_active":"2023-10-31T12:04:04.743843+0000","last_became_peered":"2023-10-31T12:04:04.743843+0000","last_unstale":"2023-10-31T12:04:04.746534+0000","last_undegraded":"2023-10-31T12:04:04.746534+0000","last_fullsized":"2023-10-31T12:04:04.746534+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-31T12:04:04.763405+0000","last_change":"2023-10-31T12:04:04.763502+0000","last_active":"2023-10-31T12:04:04.763405+0000","last_peered":"2023-10-31T12:04:04.763405+0000","last_clean":"2023-10-31T12:04:03.624440+0000","last_became_active":"2023-10-31T12:04:04.761273+0000","last_became_peered":"2023-10-31T12:04:04.761273+0000","last_unstale":"2023-10-31T12:04:04.763405+0000","last_undegraded":"2023-10-31T12:04:04.763405+0000","last_fullsized":"2023-10-31T12:04:04.763405+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-31T12:04:04.733479+0000","last_change":"2023-10-31T12:04:04.733592+0000","last_active":"2023-10-31T12:04:04.733479+0000","last_peered":"2023-10-31T12:04:04.733479+0000","last_clean":"2023-10-31T12:04:03.624440+0000","last_became_active":"2023-10-31T12:04:04.731421+0000","last_became_peered":"2023-10-31T12:04:04.731421+0000","last_unstale":"2023-10-31T12:04:04.733479+0000","last_undegraded":"2023-10-31T12:04:04.733479+0000","last_fullsized":"2023-10-31T12:04:04.733479+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.651993+0000","last_change":"2023-10-31T12:04:04.750506+0000","last_active":"2023-10-31T12:04:05.651993+0000","last_peered":"2023-10-31T12:04:05.651993+0000","last_clean":"2023-10-31T12:04:05.651993+0000","last_became_active":"2023-10-31T12:04:04.748190+0000","last_became_peered":"2023-10-31T12:04:04.748190+0000","last_unstale":"2023-10-31T12:04:05.651993+0000","last_undegraded":"2023-10-31T12:04:05.651993+0000","last_fullsized":"2023-10-31T12:04:05.651993+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.671526+0000","last_change":"2023-10-31T12:04:04.753886+0000","last_active":"2023-10-31T12:04:05.671526+0000","last_peered":"2023-10-31T12:04:05.671526+0000","last_clean":"2023-10-31T12:04:05.671526+0000","last_became_active":"2023-10-31T12:04:04.747666+0000","last_became_peered":"2023-10-31T12:04:04.747666+0000","last_unstale":"2023-10-31T12:04:05.671526+0000","last_undegraded":"2023-10-31T12:04:05.671526+0000","last_fullsized":"2023-10-31T12:04:05.671526+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26868,"kb_used_data":44,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344972,"statfs":{"total":96636764160,"available":96609251328,"internally_reserved":0,"allocated":45056,"data_stored":12715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":110592,"data_stored":605394,"data_compressed":29931,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":106496,"data_stored":603083,"data_compressed":29931,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-31T12:04:06.609 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-31T12:04:06.720 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.716+0000 7fcfb835e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:06.721+0000 7fcfb835e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:06.892 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2023-10-31T12:04:06.904 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-31T12:04:06.614845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80740,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034780,"statfs":{"total":289910292480,"available":289827614720,"internally_reserved":0,"allocated":262144,"data_stored":1221192,"data_compressed":59862,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.708967"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.651993+0000","last_change":"2023-10-31T12:04:04.717456+0000","last_active":"2023-10-31T12:04:05.651993+0000","last_peered":"2023-10-31T12:04:05.651993+0000","last_clean":"2023-10-31T12:04:05.651993+0000","last_became_active":"2023-10-31T12:04:04.715686+0000","last_became_peered":"2023-10-31T12:04:04.715686+0000","last_unstale":"2023-10-31T12:04:05.651993+0000","last_undegraded":"2023-10-31T12:04:05.651993+0000","last_fullsized":"2023-10-31T12:04:05.651993+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.653443+0000","last_change":"2023-10-31T12:04:04.789622+0000","last_active":"2023-10-31T12:04:05.653443+0000","last_peered":"2023-10-31T12:04:05.653443+0000","last_clean":"2023-10-31T12:04:05.653443+0000","last_became_active":"2023-10-31T12:04:04.787675+0000","last_became_peered":"2023-10-31T12:04:04.787675+0000","last_unstale":"2023-10-31T12:04:05.653443+0000","last_undegraded":"2023-10-31T12:04:05.653443+0000","last_fullsized":"2023-10-31T12:04:05.653443+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.652066+0000","last_change":"2023-10-31T12:04:04.736124+0000","last_active":"2023-10-31T12:04:05.652066+0000","last_peered":"2023-10-31T12:04:05.652066+0000","last_clean":"2023-10-31T12:04:05.652066+0000","last_became_active":"2023-10-31T12:04:04.727697+0000","last_became_peered":"2023-10-31T12:04:04.727697+0000","last_unstale":"2023-10-31T12:04:05.652066+0000","last_undegraded":"2023-10-31T12:04:05.652066+0000","last_fullsized":"2023-10-31T12:04:05.652066+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.653757+0000","last_change":"2023-10-31T12:04:04.772694+0000","last_active":"2023-10-31T12:04:05.653757+0000","last_peered":"2023-10-31T12:04:05.653757+0000","last_clean":"2023-10-31T12:04:05.653757+0000","last_became_active":"2023-10-31T12:04:04.768360+0000","last_became_peered":"2023-10-31T12:04:04.768360+0000","last_unstale":"2023-10-31T12:04:05.653757+0000","last_undegraded":"2023-10-31T12:04:05.653757+0000","last_fullsized":"2023-10-31T12:04:05.653757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-31T12:04:04.746534+0000","last_change":"2023-10-31T12:04:04.746754+0000","last_active":"2023-10-31T12:04:04.746534+0000","last_peered":"2023-10-31T12:04:04.746534+0000","last_clean":"2023-10-31T12:04:03.624440+0000","last_became_active":"2023-10-31T12:04:04.743843+0000","last_became_peered":"2023-10-31T12:04:04.743843+0000","last_unstale":"2023-10-31T12:04:04.746534+0000","last_undegraded":"2023-10-31T12:04:04.746534+0000","last_fullsized":"2023-10-31T12:04:04.746534+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-31T12:04:04.763405+0000","last_change":"2023-10-31T12:04:04.763502+0000","last_active":"2023-10-31T12:04:04.763405+0000","last_peered":"2023-10-31T12:04:04.763405+0000","last_clean":"2023-10-31T12:04:03.624440+0000","last_became_active":"2023-10-31T12:04:04.761273+0000","last_became_peered":"2023-10-31T12:04:04.761273+0000","last_unstale":"2023-10-31T12:04:04.763405+0000","last_undegraded":"2023-10-31T12:04:04.763405+0000","last_fullsized":"2023-10-31T12:04:04.763405+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-31T12:04:04.733479+0000","last_change":"2023-10-31T12:04:04.733592+0000","last_active":"2023-10-31T12:04:04.733479+0000","last_peered":"2023-10-31T12:04:04.733479+0000","last_clean":"2023-10-31T12:04:03.624440+0000","last_became_active":"2023-10-31T12:04:04.731421+0000","last_became_peered":"2023-10-31T12:04:04.731421+0000","last_unstale":"2023-10-31T12:04:04.733479+0000","last_undegraded":"2023-10-31T12:04:04.733479+0000","last_fullsized":"2023-10-31T12:04:04.733479+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.651993+0000","last_change":"2023-10-31T12:04:04.750506+0000","last_active":"2023-10-31T12:04:05.651993+0000","last_peered":"2023-10-31T12:04:05.651993+0000","last_clean":"2023-10-31T12:04:05.651993+0000","last_became_active":"2023-10-31T12:04:04.748190+0000","last_became_peered":"2023-10-31T12:04:04.748190+0000","last_unstale":"2023-10-31T12:04:05.651993+0000","last_undegraded":"2023-10-31T12:04:05.651993+0000","last_fullsized":"2023-10-31T12:04:05.651993+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T12:04:05.671526+0000","last_change":"2023-10-31T12:04:04.753886+0000","last_active":"2023-10-31T12:04:05.671526+0000","last_peered":"2023-10-31T12:04:05.671526+0000","last_clean":"2023-10-31T12:04:05.671526+0000","last_became_active":"2023-10-31T12:04:04.747666+0000","last_became_peered":"2023-10-31T12:04:04.747666+0000","last_unstale":"2023-10-31T12:04:05.671526+0000","last_undegraded":"2023-10-31T12:04:05.671526+0000","last_fullsized":"2023-10-31T12:04:05.671526+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:04:03.624440+0000","last_clean_scrub_stamp":"2023-10-31T12:04:03.624440+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26868,"kb_used_data":44,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344972,"statfs":{"total":96636764160,"available":96609251328,"internally_reserved":0,"allocated":45056,"data_stored":12715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":110592,"data_stored":605394,"data_compressed":29931,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":106496,"data_stored":603083,"data_compressed":29931,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-31T12:04:06.905 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-31T12:04:06.905 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-31T12:04:06.905 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-31T12:04:06.906 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-31T12:04:07.011 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:07.008+0000 7f24a1224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:04:07.012+0000 7f24a1224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:04:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:07.222 INFO:teuthology.orchestra.run.smithi161.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-31T12:04:07.234 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-31T12:04:07.235 INFO:teuthology.run_tasks:Running task cbt... 2023-10-31T12:04:07.249 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi161.front.sepia.ceph.com', 'osds': ['ubuntu@smithi161.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi161.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi161.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi161.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-10-31T12:04:07.249 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-10-31T12:04:07.266 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:04:07.267 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-31T12:04:07.318 DEBUG:teuthology.orchestra.run.smithi161:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-10-31T12:04:07.373 INFO:teuthology.orchestra.run.smithi161.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-10-31T12:04:08.062 INFO:tasks.cbt:Installing collectl 2023-10-31T12:04:08.062 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-10-31T12:04:08.167 INFO:teuthology.orchestra.run.smithi161.stderr:--2023-10-31 12:04:08-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-10-31T12:04:08.428 INFO:teuthology.orchestra.run.smithi161.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2023-10-31T12:04:08.464 INFO:teuthology.orchestra.run.smithi161.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-10-31T12:04:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-31T12:04:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQO04Jwvb25tXj7CgXr9IJ7CwhWRbt4_z0nOE3sfhLOZ7n-NYWnZc3kjBKwa6WPzmmNUtVRgSabVUy2DigeI3-2jOWQ%3D%3D&use_mirror=cytranet&r= [following] 2023-10-31T12:04:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:--2023-10-31 12:04:08-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQO04Jwvb25tXj7CgXr9IJ7CwhWRbt4_z0nOE3sfhLOZ7n-NYWnZc3kjBKwa6WPzmmNUtVRgSabVUy2DigeI3-2jOWQ%3D%3D&use_mirror=cytranet&r= 2023-10-31T12:04:08.792 INFO:teuthology.orchestra.run.smithi161.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-10-31T12:04:08.891 INFO:teuthology.orchestra.run.smithi161.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-10-31T12:04:09.239 INFO:teuthology.orchestra.run.smithi161.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-31T12:04:09.240 INFO:teuthology.orchestra.run.smithi161.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-10-31T12:04:09.240 INFO:teuthology.orchestra.run.smithi161.stderr:--2023-10-31 12:04:09-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-10-31T12:04:09.269 INFO:teuthology.orchestra.run.smithi161.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2023-10-31T12:04:09.355 INFO:teuthology.orchestra.run.smithi161.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2023-10-31T12:04:09.718 INFO:teuthology.orchestra.run.smithi161.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-31T12:04:09.718 INFO:teuthology.orchestra.run.smithi161.stderr:Length: 620693 (606K) [application/x-gzip] 2023-10-31T12:04:09.718 INFO:teuthology.orchestra.run.smithi161.stderr:Saving to: ‘collectl.tar.gz’ 2023-10-31T12:04:09.719 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:04:09.797 INFO:teuthology.orchestra.run.smithi161.stderr: 0K .......... .......... .......... .......... .......... 8% 630K 1s 2023-10-31T12:04:09.870 INFO:teuthology.orchestra.run.smithi161.stderr: 50K .......... .......... .......... .......... .......... 16% 685K 1s 2023-10-31T12:04:09.871 INFO:teuthology.orchestra.run.smithi161.stderr: 100K .......... .......... .......... .......... .......... 24% 41.4M 0s 2023-10-31T12:04:09.941 INFO:teuthology.orchestra.run.smithi161.stderr: 150K .......... .......... .......... .......... .......... 32% 715K 0s 2023-10-31T12:04:09.953 INFO:teuthology.orchestra.run.smithi161.stderr: 200K .......... .......... .......... .......... .......... 41% 3.94M 0s 2023-10-31T12:04:09.955 INFO:teuthology.orchestra.run.smithi161.stderr: 250K .......... .......... .......... .......... .......... 49% 25.6M 0s 2023-10-31T12:04:09.957 INFO:teuthology.orchestra.run.smithi161.stderr: 300K .......... .......... .......... .......... .......... 57% 21.4M 0s 2023-10-31T12:04:10.052 INFO:teuthology.orchestra.run.smithi161.stderr: 350K .......... .......... .......... .......... .......... 65% 532K 0s 2023-10-31T12:04:10.062 INFO:teuthology.orchestra.run.smithi161.stderr: 400K .......... .......... .......... .......... .......... 74% 4.72M 0s 2023-10-31T12:04:10.065 INFO:teuthology.orchestra.run.smithi161.stderr: 450K .......... .......... .......... .......... .......... 82% 17.6M 0s 2023-10-31T12:04:10.066 INFO:teuthology.orchestra.run.smithi161.stderr: 500K .......... .......... .......... .......... .......... 90% 44.8M 0s 2023-10-31T12:04:10.069 INFO:teuthology.orchestra.run.smithi161.stderr: 550K .......... .......... .......... .......... .......... 98% 21.3M 0s 2023-10-31T12:04:10.069 INFO:teuthology.orchestra.run.smithi161.stderr: 600K ...... 100% 11.4T=0.4s 2023-10-31T12:04:10.070 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:04:10.070 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31 12:04:10 (1.69 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-10-31T12:04:10.070 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:04:10.109 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/ 2023-10-31T12:04:10.109 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/collectl.conf 2023-10-31T12:04:10.109 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/ 2023-10-31T12:04:10.109 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/SlowProc.html 2023-10-31T12:04:10.110 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-10-31T12:04:10.110 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/WhySummary.html 2023-10-31T12:04:10.110 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Socket.html 2023-10-31T12:04:10.110 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/colmux.html 2023-10-31T12:04:10.110 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Tutorial.html 2023-10-31T12:04:10.111 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Gexpr.html 2023-10-31T12:04:10.111 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Import.html 2023-10-31T12:04:10.111 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/style.css 2023-10-31T12:04:10.111 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-10-31T12:04:10.111 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-10-31T12:04:10.112 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/FileNaming.html 2023-10-31T12:04:10.112 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/TheMath.html 2023-10-31T12:04:10.112 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Environmental.html 2023-10-31T12:04:10.113 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Process.html 2023-10-31T12:04:10.113 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-10-31T12:04:10.113 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/HiResTime.html 2023-10-31T12:04:10.114 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-10-31T12:04:10.114 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/slub.jpg 2023-10-31T12:04:10.114 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Lustre.html 2023-10-31T12:04:10.114 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Documentation.html 2023-10-31T12:04:10.115 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Inodes.html 2023-10-31T12:04:10.115 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Graphite.html 2023-10-31T12:04:10.115 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/InputFiles.html 2023-10-31T12:04:10.115 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Export.html 2023-10-31T12:04:10.115 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Interrupts.html 2023-10-31T12:04:10.116 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Startup.html 2023-10-31T12:04:10.116 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Performance.html 2023-10-31T12:04:10.116 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Messages.html 2023-10-31T12:04:10.117 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Logging.html 2023-10-31T12:04:10.117 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Data-detail.html 2023-10-31T12:04:10.117 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/CPUs.html 2023-10-31T12:04:10.118 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Matrix.html 2023-10-31T12:04:10.118 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Data-brief.html 2023-10-31T12:04:10.118 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-10-31T12:04:10.119 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Misc.html 2023-10-31T12:04:10.119 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Examples.html 2023-10-31T12:04:10.119 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/OpenStack.html 2023-10-31T12:04:10.120 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-10-31T12:04:10.120 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-10-31T12:04:10.120 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-10-31T12:04:10.120 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-10-31T12:04:10.121 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Data.html 2023-10-31T12:04:10.121 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Exceptions.html 2023-10-31T12:04:10.121 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-10-31T12:04:10.122 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-10-31T12:04:10.122 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-10-31T12:04:10.122 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-10-31T12:04:10.122 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Playback.html 2023-10-31T12:04:10.123 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Hello.html 2023-10-31T12:04:10.123 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Sockets.html 2023-10-31T12:04:10.123 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Network.html 2023-10-31T12:04:10.124 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Infiniband.html 2023-10-31T12:04:10.124 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Memory.html 2023-10-31T12:04:10.124 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-10-31T12:04:10.124 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/Disks.html 2023-10-31T12:04:10.125 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-10-31T12:04:10.125 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/service/ 2023-10-31T12:04:10.125 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/service/collectl.service 2023-10-31T12:04:10.125 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/initd/ 2023-10-31T12:04:10.126 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/initd/collectl-suse 2023-10-31T12:04:10.126 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/initd/collectl-debian 2023-10-31T12:04:10.126 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/initd/collectl-generic 2023-10-31T12:04:10.127 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/initd/collectl 2023-10-31T12:04:10.127 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/misc.ph 2023-10-31T12:04:10.128 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/gexpr.ph 2023-10-31T12:04:10.128 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/statsd.ph 2023-10-31T12:04:10.128 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/envrules.std 2023-10-31T12:04:10.129 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/hello.ph 2023-10-31T12:04:10.129 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/vmsum.ph 2023-10-31T12:04:10.129 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/proctree.ph 2023-10-31T12:04:10.129 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/README-WINDOWS 2023-10-31T12:04:10.130 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/formatit.ph 2023-10-31T12:04:10.130 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/colmux 2023-10-31T12:04:10.130 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/INSTALL 2023-10-31T12:04:10.131 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/graphite.ph 2023-10-31T12:04:10.131 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/RELEASE-collectl 2023-10-31T12:04:10.133 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/vmstat.ph 2023-10-31T12:04:10.133 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/README 2023-10-31T12:04:10.133 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/man1/ 2023-10-31T12:04:10.133 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/man1/collectl.1 2023-10-31T12:04:10.133 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/man1/colmux.1 2023-10-31T12:04:10.134 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/lexpr.ph 2023-10-31T12:04:10.134 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/client.pl 2023-10-31T12:04:10.134 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/collectl 2023-10-31T12:04:10.134 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/COPYING 2023-10-31T12:04:10.134 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/vnet.ph 2023-10-31T12:04:10.135 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/GPL 2023-10-31T12:04:10.135 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/UNINSTALL 2023-10-31T12:04:10.135 INFO:teuthology.orchestra.run.smithi161.stdout:collectl-4.3.1/ARTISTIC 2023-10-31T12:04:10.168 INFO:teuthology.orchestra.run.smithi161.stderr:./INSTALL: line 90: chkconfig: command not found 2023-10-31T12:04:10.169 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-10-31T12:04:10.718 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:27 ago on Tue 31 Oct 2023 12:00:43 PM UTC. 2023-10-31T12:04:10.810 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-10-31T12:04:10.813 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-10-31T12:04:10.868 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2023-10-31T12:04:10.868 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:04:10.868 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2023-10-31T12:04:10.869 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:04:10.869 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2023-10-31T12:04:10.869 INFO:teuthology.orchestra.run.smithi161.stdout: librbd-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 23 k 2023-10-31T12:04:10.869 INFO:teuthology.orchestra.run.smithi161.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-10-31T12:04:10.869 INFO:teuthology.orchestra.run.smithi161.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-10-31T12:04:10.870 INFO:teuthology.orchestra.run.smithi161.stdout: perf x86_64 5.14.0-378.el9 appstream 7.3 M 2023-10-31T12:04:10.870 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2023-10-31T12:04:10.870 INFO:teuthology.orchestra.run.smithi161.stdout: libradospp-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 30 k 2023-10-31T12:04:10.870 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:10.870 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2023-10-31T12:04:10.871 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2023-10-31T12:04:10.871 INFO:teuthology.orchestra.run.smithi161.stdout:Install 5 Packages 2023-10-31T12:04:10.871 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:10.872 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 7.5 M 2023-10-31T12:04:10.872 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 11 M 2023-10-31T12:04:10.872 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2023-10-31T12:04:11.290 INFO:teuthology.orchestra.run.smithi161.stdout:(1/5): librbd-devel-18.0.0-6792.ge83bf482.el9.x 131 kB/s | 23 kB 00:00 2023-10-31T12:04:11.349 INFO:teuthology.orchestra.run.smithi161.stdout:(2/5): libradospp-devel-18.0.0-6792.ge83bf482.e 128 kB/s | 30 kB 00:00 2023-10-31T12:04:11.524 INFO:teuthology.orchestra.run.smithi161.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 87 kB/s | 13 kB 00:00 2023-10-31T12:04:11.565 INFO:teuthology.orchestra.run.smithi161.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 431 kB/s | 118 kB 00:00 2023-10-31T12:04:11.824 INFO:teuthology.orchestra.run.smithi161.stdout:(5/5): perf-5.14.0-378.el9.x86_64.rpm 10 MB/s | 7.3 MB 00:00 2023-10-31T12:04:11.827 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:04:11.827 INFO:teuthology.orchestra.run.smithi161.stdout:Total 7.8 MB/s | 7.5 MB 00:00 2023-10-31T12:04:11.904 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2023-10-31T12:04:11.926 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2023-10-31T12:04:11.926 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2023-10-31T12:04:12.124 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2023-10-31T12:04:12.125 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2023-10-31T12:04:12.477 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2023-10-31T12:04:12.551 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-10-31T12:04:12.590 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-10-31T12:04:12.630 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradospp-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 3/5 2023-10-31T12:04:12.977 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librbd-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 4/5 2023-10-31T12:04:13.093 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : perf-5.14.0-378.el9.x86_64 5/5 2023-10-31T12:04:14.163 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: perf-5.14.0-378.el9.x86_64 5/5 2023-10-31T12:04:14.163 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradospp-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/5 2023-10-31T12:04:14.163 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2/5 2023-10-31T12:04:14.164 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : perf-5.14.0-378.el9.x86_64 3/5 2023-10-31T12:04:14.164 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-10-31T12:04:14.524 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-10-31T12:04:14.525 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:14.525 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2023-10-31T12:04:14.525 INFO:teuthology.orchestra.run.smithi161.stdout: libradospp-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:04:14.526 INFO:teuthology.orchestra.run.smithi161.stdout: librbd-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:04:14.526 INFO:teuthology.orchestra.run.smithi161.stdout: pdsh-2.34-7.el9.x86_64 2023-10-31T12:04:14.526 INFO:teuthology.orchestra.run.smithi161.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-10-31T12:04:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: perf-5.14.0-378.el9.x86_64 2023-10-31T12:04:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:14.527 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2023-10-31T12:04:14.934 INFO:tasks.cbt:benchmark: librbdfio 2023-10-31T12:04:14.935 DEBUG:teuthology.orchestra.run.smithi161:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2023-10-31T12:04:14.950 INFO:teuthology.orchestra.run.smithi161.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2023-10-31T12:04:17.081 DEBUG:teuthology.orchestra.run.smithi161:> cd /home/ubuntu/cephtest/fio && ./configure && make 2023-10-31T12:04:17.655 INFO:teuthology.orchestra.run.smithi161.stdout:Operating system Linux 2023-10-31T12:04:17.656 INFO:teuthology.orchestra.run.smithi161.stdout:CPU x86_64 2023-10-31T12:04:17.656 INFO:teuthology.orchestra.run.smithi161.stdout:Big endian no 2023-10-31T12:04:17.656 INFO:teuthology.orchestra.run.smithi161.stdout:Compiler gcc 2023-10-31T12:04:17.656 INFO:teuthology.orchestra.run.smithi161.stdout:Cross compile no 2023-10-31T12:04:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:04:17.657 INFO:teuthology.orchestra.run.smithi161.stdout:Static build no 2023-10-31T12:04:17.759 INFO:teuthology.orchestra.run.smithi161.stdout:Wordsize 64 2023-10-31T12:04:17.766 INFO:teuthology.orchestra.run.smithi161.stdout:zlib no 2023-10-31T12:04:17.844 INFO:teuthology.orchestra.run.smithi161.stdout:fcntl(F_FULLFSYNC) no 2023-10-31T12:04:17.858 INFO:teuthology.orchestra.run.smithi161.stdout:Linux AIO support no 2023-10-31T12:04:17.858 INFO:teuthology.orchestra.run.smithi161.stdout:Linux AIO support rw flags no 2023-10-31T12:04:17.908 INFO:teuthology.orchestra.run.smithi161.stdout:POSIX AIO support yes 2023-10-31T12:04:17.909 INFO:teuthology.orchestra.run.smithi161.stdout:POSIX AIO support needs -lrt no 2023-10-31T12:04:17.939 INFO:teuthology.orchestra.run.smithi161.stdout:POSIX AIO fsync yes 2023-10-31T12:04:17.968 INFO:teuthology.orchestra.run.smithi161.stdout:POSIX pshared support yes 2023-10-31T12:04:18.023 INFO:teuthology.orchestra.run.smithi161.stdout:pthread_condattr_setclock() yes 2023-10-31T12:04:18.062 INFO:teuthology.orchestra.run.smithi161.stdout:pthread_sigmask() yes 2023-10-31T12:04:18.098 INFO:teuthology.orchestra.run.smithi161.stdout:pthread_getaffinity_np() yes 2023-10-31T12:04:18.107 INFO:teuthology.orchestra.run.smithi161.stdout:Solaris AIO support no 2023-10-31T12:04:18.155 INFO:teuthology.orchestra.run.smithi161.stdout:__sync_fetch_and_add yes 2023-10-31T12:04:18.195 INFO:teuthology.orchestra.run.smithi161.stdout:__sync_synchronize yes 2023-10-31T12:04:18.223 INFO:teuthology.orchestra.run.smithi161.stdout:__sync_val_compare_and_swap yes 2023-10-31T12:04:18.230 INFO:teuthology.orchestra.run.smithi161.stdout:libverbs no 2023-10-31T12:04:18.262 INFO:teuthology.orchestra.run.smithi161.stdout:rdmacm no 2023-10-31T12:04:18.269 INFO:teuthology.orchestra.run.smithi161.stdout:librpma no 2023-10-31T12:04:18.279 INFO:teuthology.orchestra.run.smithi161.stdout:libprotobuf_c no 2023-10-31T12:04:18.308 INFO:teuthology.orchestra.run.smithi161.stdout:asprintf() yes 2023-10-31T12:04:18.338 INFO:teuthology.orchestra.run.smithi161.stdout:vasprintf() yes 2023-10-31T12:04:18.369 INFO:teuthology.orchestra.run.smithi161.stdout:Linux fallocate yes 2023-10-31T12:04:18.400 INFO:teuthology.orchestra.run.smithi161.stdout:POSIX fadvise yes 2023-10-31T12:04:18.431 INFO:teuthology.orchestra.run.smithi161.stdout:POSIX fallocate yes 2023-10-31T12:04:18.461 INFO:teuthology.orchestra.run.smithi161.stdout:sched_setaffinity(3 arg) yes 2023-10-31T12:04:18.462 INFO:teuthology.orchestra.run.smithi161.stdout:sched_setaffinity(2 arg) no 2023-10-31T12:04:18.491 INFO:teuthology.orchestra.run.smithi161.stdout:clock_gettime yes 2023-10-31T12:04:18.521 INFO:teuthology.orchestra.run.smithi161.stdout:CLOCK_MONOTONIC yes 2023-10-31T12:04:18.560 INFO:teuthology.orchestra.run.smithi161.stdout:clockid_t yes 2023-10-31T12:04:18.602 INFO:teuthology.orchestra.run.smithi161.stdout:gettimeofday yes 2023-10-31T12:04:18.637 INFO:teuthology.orchestra.run.smithi161.stdout:fdatasync yes 2023-10-31T12:04:18.667 INFO:teuthology.orchestra.run.smithi161.stdout:pipe() yes 2023-10-31T12:04:18.697 INFO:teuthology.orchestra.run.smithi161.stdout:pipe2() yes 2023-10-31T12:04:18.728 INFO:teuthology.orchestra.run.smithi161.stdout:pread() yes 2023-10-31T12:04:18.798 INFO:teuthology.orchestra.run.smithi161.stdout:sync_file_range yes 2023-10-31T12:04:18.805 INFO:teuthology.orchestra.run.smithi161.stdout:ASharedMemory_create no 2023-10-31T12:04:18.836 INFO:teuthology.orchestra.run.smithi161.stdout:EXT4 move extent yes 2023-10-31T12:04:18.867 INFO:teuthology.orchestra.run.smithi161.stdout:Linux splice(2) yes 2023-10-31T12:04:18.873 INFO:teuthology.orchestra.run.smithi161.stdout:libnuma no 2023-10-31T12:04:18.902 INFO:teuthology.orchestra.run.smithi161.stdout:strsep yes 2023-10-31T12:04:18.933 INFO:teuthology.orchestra.run.smithi161.stdout:strcasestr yes 2023-10-31T12:04:18.946 INFO:teuthology.orchestra.run.smithi161.stdout:strlcat no 2023-10-31T12:04:18.980 INFO:teuthology.orchestra.run.smithi161.stdout:getopt_long_only() yes 2023-10-31T12:04:19.039 INFO:teuthology.orchestra.run.smithi161.stdout:inet_aton yes 2023-10-31T12:04:19.069 INFO:teuthology.orchestra.run.smithi161.stdout:socklen_t yes 2023-10-31T12:04:19.101 INFO:teuthology.orchestra.run.smithi161.stdout:__thread yes 2023-10-31T12:04:19.132 INFO:teuthology.orchestra.run.smithi161.stdout:RUSAGE_THREAD yes 2023-10-31T12:04:19.161 INFO:teuthology.orchestra.run.smithi161.stdout:SCHED_IDLE yes 2023-10-31T12:04:19.195 INFO:teuthology.orchestra.run.smithi161.stdout:TCP_NODELAY yes 2023-10-31T12:04:19.228 INFO:teuthology.orchestra.run.smithi161.stdout:Net engine window_size yes 2023-10-31T12:04:19.262 INFO:teuthology.orchestra.run.smithi161.stdout:TCP_MAXSEG yes 2023-10-31T12:04:19.290 INFO:teuthology.orchestra.run.smithi161.stdout:RLIMIT_MEMLOCK yes 2023-10-31T12:04:19.324 INFO:teuthology.orchestra.run.smithi161.stdout:pwritev/preadv yes 2023-10-31T12:04:19.356 INFO:teuthology.orchestra.run.smithi161.stdout:pwritev2/preadv2 yes 2023-10-31T12:04:19.393 INFO:teuthology.orchestra.run.smithi161.stdout:IPv6 helpers yes 2023-10-31T12:04:19.409 INFO:teuthology.orchestra.run.smithi161.stdout:http engine no 2023-10-31T12:04:19.501 INFO:teuthology.orchestra.run.smithi161.stdout:Rados engine yes 2023-10-31T12:04:19.605 INFO:teuthology.orchestra.run.smithi161.stdout:Rados Block Device engine yes 2023-10-31T12:04:19.716 INFO:teuthology.orchestra.run.smithi161.stdout:rbd_poll yes 2023-10-31T12:04:19.819 INFO:teuthology.orchestra.run.smithi161.stdout:rbd_invalidate_cache yes 2023-10-31T12:04:19.849 INFO:teuthology.orchestra.run.smithi161.stdout:setvbuf yes 2023-10-31T12:04:19.856 INFO:teuthology.orchestra.run.smithi161.stdout:Gluster API engine no 2023-10-31T12:04:19.871 INFO:teuthology.orchestra.run.smithi161.stdout:s390_z196_facilities no 2023-10-31T12:04:19.872 INFO:teuthology.orchestra.run.smithi161.stdout:HDFS engine no 2023-10-31T12:04:19.927 INFO:teuthology.orchestra.run.smithi161.stdout:MTD yes 2023-10-31T12:04:19.934 INFO:teuthology.orchestra.run.smithi161.stdout:libpmem no 2023-10-31T12:04:19.935 INFO:teuthology.orchestra.run.smithi161.stdout:libpmem1_5 no 2023-10-31T12:04:19.941 INFO:teuthology.orchestra.run.smithi161.stdout:libpmem2 no 2023-10-31T12:04:19.941 INFO:teuthology.orchestra.run.smithi161.stdout:PMDK dev-dax engine no 2023-10-31T12:04:19.942 INFO:teuthology.orchestra.run.smithi161.stdout:PMDK libpmem engine no 2023-10-31T12:04:19.948 INFO:teuthology.orchestra.run.smithi161.stdout:DDN's Infinite Memory Engine no 2023-10-31T12:04:19.948 INFO:teuthology.orchestra.run.smithi161.stdout:iscsi engine no 2023-10-31T12:04:19.949 INFO:teuthology.orchestra.run.smithi161.stdout:NBD engine no 2023-10-31T12:04:19.957 INFO:teuthology.orchestra.run.smithi161.stdout:DAOS File System (dfs) Engine no 2023-10-31T12:04:19.966 INFO:teuthology.orchestra.run.smithi161.stdout:NFS engine no 2023-10-31T12:04:19.966 INFO:teuthology.orchestra.run.smithi161.stdout:lex/yacc for arithmetic no 2023-10-31T12:04:19.998 INFO:teuthology.orchestra.run.smithi161.stdout:getmntent yes 2023-10-31T12:04:20.017 INFO:teuthology.orchestra.run.smithi161.stdout:getmntinfo no 2023-10-31T12:04:20.074 INFO:teuthology.orchestra.run.smithi161.stdout:Static Assert yes 2023-10-31T12:04:20.102 INFO:teuthology.orchestra.run.smithi161.stdout:bool yes 2023-10-31T12:04:20.135 INFO:teuthology.orchestra.run.smithi161.stdout:strndup yes 2023-10-31T12:04:20.142 INFO:teuthology.orchestra.run.smithi161.stdout:Valgrind headers no 2023-10-31T12:04:20.183 INFO:teuthology.orchestra.run.smithi161.stdout:Zoned block device support yes 2023-10-31T12:04:20.211 INFO:teuthology.orchestra.run.smithi161.stdout:Zoned block device capacity yes 2023-10-31T12:04:20.218 INFO:teuthology.orchestra.run.smithi161.stdout:libzbc engine no 2023-10-31T12:04:20.256 INFO:teuthology.orchestra.run.smithi161.stdout:NVMe uring command support yes 2023-10-31T12:04:20.270 INFO:teuthology.orchestra.run.smithi161.stdout:xnvme engine no 2023-10-31T12:04:20.284 INFO:teuthology.orchestra.run.smithi161.stdout:libblkio engine no 2023-10-31T12:04:20.284 INFO:teuthology.orchestra.run.smithi161.stdout:march_armv8_a_crc_crypto no 2023-10-31T12:04:20.285 INFO:teuthology.orchestra.run.smithi161.stdout:cuda no 2023-10-31T12:04:20.285 INFO:teuthology.orchestra.run.smithi161.stdout:libcufile no 2023-10-31T12:04:20.311 INFO:teuthology.orchestra.run.smithi161.stdout:Build march=native yes 2023-10-31T12:04:20.318 INFO:teuthology.orchestra.run.smithi161.stdout:CUnit no 2023-10-31T12:04:20.347 INFO:teuthology.orchestra.run.smithi161.stdout:__kernel_rwf_t yes 2023-10-31T12:04:20.373 INFO:teuthology.orchestra.run.smithi161.stdout:-Wimplicit-fallthrough=2 yes 2023-10-31T12:04:20.404 INFO:teuthology.orchestra.run.smithi161.stdout:-Wno-stringop-truncation yes 2023-10-31T12:04:20.434 INFO:teuthology.orchestra.run.smithi161.stdout:MADV_HUGEPAGE yes 2023-10-31T12:04:20.464 INFO:teuthology.orchestra.run.smithi161.stdout:gettid yes 2023-10-31T12:04:20.499 INFO:teuthology.orchestra.run.smithi161.stdout:statx(2)/libc yes 2023-10-31T12:04:20.532 INFO:teuthology.orchestra.run.smithi161.stdout:statx(2)/syscall yes 2023-10-31T12:04:20.534 INFO:teuthology.orchestra.run.smithi161.stdout:Windows PDB generation no 2023-10-31T12:04:20.565 INFO:teuthology.orchestra.run.smithi161.stdout:timerfd_create yes 2023-10-31T12:04:20.566 INFO:teuthology.orchestra.run.smithi161.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2023-10-31T12:04:20.567 INFO:teuthology.orchestra.run.smithi161.stdout:Lib-based ioengines dynamic no 2023-10-31T12:04:20.628 INFO:teuthology.orchestra.run.smithi161.stdout:TCMalloc support yes 2023-10-31T12:04:20.629 INFO:teuthology.orchestra.run.smithi161.stdout:seed_buckets 4 2023-10-31T12:04:20.711 INFO:teuthology.orchestra.run.smithi161.stderr:FIO_VERSION = fio-3.36-9-g7a72 2023-10-31T12:04:20.750 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc16.o 2023-10-31T12:04:20.838 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc32.o 2023-10-31T12:04:20.873 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc32c-arm64.o 2023-10-31T12:04:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc32c-intel.o 2023-10-31T12:04:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc32c.o 2023-10-31T12:04:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc64.o 2023-10-31T12:04:21.071 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crc7.o 2023-10-31T12:04:21.103 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/crct10dif_common.o 2023-10-31T12:04:21.136 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/fnv.o 2023-10-31T12:04:21.177 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/md5.o 2023-10-31T12:04:21.280 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/murmur3.o 2023-10-31T12:04:21.321 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/sha1.o 2023-10-31T12:04:21.507 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/sha256.o 2023-10-31T12:04:21.795 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/sha3.o 2023-10-31T12:04:21.937 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/sha512.o 2023-10-31T12:04:22.043 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/test.o 2023-10-31T12:04:22.204 INFO:teuthology.orchestra.run.smithi161.stdout: CC crc/xxhash.o 2023-10-31T12:04:22.297 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/axmap.o 2023-10-31T12:04:22.418 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/bloom.o 2023-10-31T12:04:22.500 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/flist_sort.o 2023-10-31T12:04:22.571 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/gauss.o 2023-10-31T12:04:22.668 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/getrusage.o 2023-10-31T12:04:22.703 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/hweight.o 2023-10-31T12:04:22.740 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/ieee754.o 2023-10-31T12:04:22.789 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/lfsr.o 2023-10-31T12:04:22.888 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/memalign.o 2023-10-31T12:04:22.931 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/memcpy.o 2023-10-31T12:04:23.073 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/mountcheck.o 2023-10-31T12:04:23.113 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/num2str.o 2023-10-31T12:04:23.179 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/output_buffer.o 2023-10-31T12:04:23.225 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/pattern.o 2023-10-31T12:04:23.366 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/prio_tree.o 2023-10-31T12:04:23.485 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/rand.o 2023-10-31T12:04:23.673 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/rbtree.o 2023-10-31T12:04:23.796 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/strntol.o 2023-10-31T12:04:23.843 INFO:teuthology.orchestra.run.smithi161.stdout: CC lib/zipf.o 2023-10-31T12:04:23.961 INFO:teuthology.orchestra.run.smithi161.stdout: CC gettime.o 2023-10-31T12:04:24.251 INFO:teuthology.orchestra.run.smithi161.stdout: CC ioengines.o 2023-10-31T12:04:24.515 INFO:teuthology.orchestra.run.smithi161.stdout: CC init.o 2023-10-31T12:04:25.286 INFO:teuthology.orchestra.run.smithi161.stdout: CC stat.o 2023-10-31T12:04:27.150 INFO:teuthology.orchestra.run.smithi161.stdout: CC log.o 2023-10-31T12:04:27.277 INFO:teuthology.orchestra.run.smithi161.stdout: CC time.o 2023-10-31T12:04:27.396 INFO:teuthology.orchestra.run.smithi161.stdout: CC filesetup.o 2023-10-31T12:04:28.087 INFO:teuthology.orchestra.run.smithi161.stdout: CC eta.o 2023-10-31T12:04:28.397 INFO:teuthology.orchestra.run.smithi161.stdout: CC verify.o 2023-10-31T12:04:29.079 INFO:teuthology.orchestra.run.smithi161.stdout: CC memory.o 2023-10-31T12:04:29.247 INFO:teuthology.orchestra.run.smithi161.stdout: CC io_u.o 2023-10-31T12:04:29.996 INFO:teuthology.orchestra.run.smithi161.stdout: CC parse.o 2023-10-31T12:04:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: CC fio_sem.o 2023-10-31T12:04:30.538 INFO:teuthology.orchestra.run.smithi161.stdout: CC rwlock.o 2023-10-31T12:04:30.621 INFO:teuthology.orchestra.run.smithi161.stdout: CC pshared.o 2023-10-31T12:04:30.686 INFO:teuthology.orchestra.run.smithi161.stdout: CC options.o 2023-10-31T12:04:31.457 INFO:teuthology.orchestra.run.smithi161.stdout: CC smalloc.o 2023-10-31T12:04:31.639 INFO:teuthology.orchestra.run.smithi161.stdout: CC filehash.o 2023-10-31T12:04:31.787 INFO:teuthology.orchestra.run.smithi161.stdout: CC profile.o 2023-10-31T12:04:31.903 INFO:teuthology.orchestra.run.smithi161.stdout: CC debug.o 2023-10-31T12:04:31.944 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/cpu.o 2023-10-31T12:04:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/mmap.o 2023-10-31T12:04:32.244 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/sync.o 2023-10-31T12:04:32.471 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/null.o 2023-10-31T12:04:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/net.o 2023-10-31T12:04:32.981 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/ftruncate.o 2023-10-31T12:04:33.075 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/fileoperations.o 2023-10-31T12:04:33.216 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/exec.o 2023-10-31T12:04:33.364 INFO:teuthology.orchestra.run.smithi161.stdout: CC server.o 2023-10-31T12:04:34.093 INFO:teuthology.orchestra.run.smithi161.stdout: CC client.o 2023-10-31T12:04:34.627 INFO:teuthology.orchestra.run.smithi161.stdout: CC iolog.o 2023-10-31T12:04:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: CC backend.o 2023-10-31T12:04:35.895 INFO:teuthology.orchestra.run.smithi161.stdout: CC libfio.o 2023-10-31T12:04:36.070 INFO:teuthology.orchestra.run.smithi161.stdout: CC flow.o 2023-10-31T12:04:36.187 INFO:teuthology.orchestra.run.smithi161.stdout: CC cconv.o 2023-10-31T12:04:36.534 INFO:teuthology.orchestra.run.smithi161.stdout: CC gettime-thread.o 2023-10-31T12:04:36.634 INFO:teuthology.orchestra.run.smithi161.stdout: CC helpers.o 2023-10-31T12:04:36.696 INFO:teuthology.orchestra.run.smithi161.stdout: CC json.o 2023-10-31T12:04:36.881 INFO:teuthology.orchestra.run.smithi161.stdout: CC idletime.o 2023-10-31T12:04:37.104 INFO:teuthology.orchestra.run.smithi161.stdout: CC td_error.o 2023-10-31T12:04:37.202 INFO:teuthology.orchestra.run.smithi161.stdout: CC profiles/tiobench.o 2023-10-31T12:04:37.296 INFO:teuthology.orchestra.run.smithi161.stdout: CC profiles/act.o 2023-10-31T12:04:37.451 INFO:teuthology.orchestra.run.smithi161.stdout: CC io_u_queue.o 2023-10-31T12:04:37.501 INFO:teuthology.orchestra.run.smithi161.stdout: CC filelock.o 2023-10-31T12:04:37.624 INFO:teuthology.orchestra.run.smithi161.stdout: CC workqueue.o 2023-10-31T12:04:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: CC rate-submit.o 2023-10-31T12:04:37.939 INFO:teuthology.orchestra.run.smithi161.stdout: CC optgroup.o 2023-10-31T12:04:37.984 INFO:teuthology.orchestra.run.smithi161.stdout: CC helper_thread.o 2023-10-31T12:04:38.160 INFO:teuthology.orchestra.run.smithi161.stdout: CC steadystate.o 2023-10-31T12:04:38.395 INFO:teuthology.orchestra.run.smithi161.stdout: CC zone-dist.o 2023-10-31T12:04:38.524 INFO:teuthology.orchestra.run.smithi161.stdout: CC zbd.o 2023-10-31T12:04:39.208 INFO:teuthology.orchestra.run.smithi161.stdout: CC dedupe.o 2023-10-31T12:04:39.349 INFO:teuthology.orchestra.run.smithi161.stdout: CC fdp.o 2023-10-31T12:04:39.487 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/posixaio.o 2023-10-31T12:04:39.616 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/falloc.o 2023-10-31T12:04:39.730 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/e4defrag.o 2023-10-31T12:04:39.861 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/splice.o 2023-10-31T12:04:40.050 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/asprintf.o 2023-10-31T12:04:40.084 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/strlcat.o 2023-10-31T12:04:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/mtd.o 2023-10-31T12:04:40.286 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/libmtd.o 2023-10-31T12:04:40.660 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/libmtd_legacy.o 2023-10-31T12:04:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: CC diskutil.o 2023-10-31T12:04:41.001 INFO:teuthology.orchestra.run.smithi161.stdout: CC fifo.o 2023-10-31T12:04:41.055 INFO:teuthology.orchestra.run.smithi161.stdout: CC blktrace.o 2023-10-31T12:04:41.383 INFO:teuthology.orchestra.run.smithi161.stdout: CC cgroup.o 2023-10-31T12:04:41.535 INFO:teuthology.orchestra.run.smithi161.stdout: CC trim.o 2023-10-31T12:04:41.645 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/sg.o 2023-10-31T12:04:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/linux-dev-lookup.o 2023-10-31T12:04:42.079 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/io_uring.o 2023-10-31T12:04:42.453 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/nvme.o 2023-10-31T12:04:42.739 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/linux-blkzoned.o 2023-10-31T12:04:42.888 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/cmdprio.o 2023-10-31T12:04:43.128 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/rados.o 2023-10-31T12:04:43.299 INFO:teuthology.orchestra.run.smithi161.stdout: CC engines/rbd.o 2023-10-31T12:04:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: CC fio.o 2023-10-31T12:04:43.567 INFO:teuthology.orchestra.run.smithi161.stdout: LINK fio 2023-10-31T12:04:43.759 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/log.o 2023-10-31T12:04:43.801 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/genzipf.o 2023-10-31T12:04:43.936 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/strcasestr.o 2023-10-31T12:04:43.959 INFO:teuthology.orchestra.run.smithi161.stdout: CC oslib/strndup.o 2023-10-31T12:04:43.982 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/fio-genzipf 2023-10-31T12:04:44.071 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/btrace2fio.o 2023-10-31T12:04:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/fio-btrace2fio 2023-10-31T12:04:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/verify-state.o 2023-10-31T12:04:44.670 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/debug.o 2023-10-31T12:04:44.705 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/fio-verify-state 2023-10-31T12:04:44.801 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/stest.o 2023-10-31T12:04:44.869 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/arch.o 2023-10-31T12:04:44.898 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/stest 2023-10-31T12:04:44.985 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/ieee754.o 2023-10-31T12:04:45.024 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/ieee754 2023-10-31T12:04:45.111 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/axmap.o 2023-10-31T12:04:45.208 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/axmap 2023-10-31T12:04:45.294 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/lfsr-test.o 2023-10-31T12:04:45.388 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/lfsr-test 2023-10-31T12:04:45.478 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/gen-rand.o 2023-10-31T12:04:45.573 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/gen-rand 2023-10-31T12:04:45.662 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/memlock.o 2023-10-31T12:04:45.722 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/memlock 2023-10-31T12:04:45.807 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/read-to-pipe-async.o 2023-10-31T12:04:45.991 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/read-to-pipe-async 2023-10-31T12:04:46.076 INFO:teuthology.orchestra.run.smithi161.stdout: CC t/io_uring.o 2023-10-31T12:04:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: LINK t/io_uring 2023-10-31T12:04:46.700 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-10-31T12:04:46.727 DEBUG:teuthology.orchestra.run.smithi161:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-31T12:04:46.966 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - Settings.cluster: 2023-10-31T12:04:46.967 INFO:teuthology.orchestra.run.smithi161.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-10-31T12:04:46.967 INFO:teuthology.orchestra.run.smithi161.stderr: 'clients': ['ubuntu@smithi161.front.sepia.ceph.com'], 2023-10-31T12:04:46.967 INFO:teuthology.orchestra.run.smithi161.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-10-31T12:04:46.967 INFO:teuthology.orchestra.run.smithi161.stderr: 'head': 'ubuntu@smithi161.front.sepia.ceph.com', 2023-10-31T12:04:46.968 INFO:teuthology.orchestra.run.smithi161.stderr: 'is_teuthology': True, 2023-10-31T12:04:46.968 INFO:teuthology.orchestra.run.smithi161.stderr: 'iterations': 1, 2023-10-31T12:04:46.968 INFO:teuthology.orchestra.run.smithi161.stderr: 'mons': ['ubuntu@smithi161.front.sepia.ceph.com'], 2023-10-31T12:04:46.968 INFO:teuthology.orchestra.run.smithi161.stderr: 'osds': ['ubuntu@smithi161.front.sepia.ceph.com'], 2023-10-31T12:04:46.969 INFO:teuthology.orchestra.run.smithi161.stderr: 'osds_per_node': 3, 2023-10-31T12:04:46.969 INFO:teuthology.orchestra.run.smithi161.stderr: 'pid_dir': '/var/run/ceph', 2023-10-31T12:04:46.969 INFO:teuthology.orchestra.run.smithi161.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2023-10-31T12:04:46.969 INFO:teuthology.orchestra.run.smithi161.stderr: 'rebuild_every_test': False, 2023-10-31T12:04:46.969 INFO:teuthology.orchestra.run.smithi161.stderr: 'rgws': {'ubuntu@smithi161.front.sepia.ceph.com': None}, 2023-10-31T12:04:46.970 INFO:teuthology.orchestra.run.smithi161.stderr: 'tmp_dir': '/tmp/cbt', 2023-10-31T12:04:46.970 INFO:teuthology.orchestra.run.smithi161.stderr: 'use_existing': True, 2023-10-31T12:04:46.970 INFO:teuthology.orchestra.run.smithi161.stderr: 'user': 'ubuntu'} 2023-10-31T12:04:46.972 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.973 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.973 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.973 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread; fi 2023-10-31T12:04:46.973 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.974 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.974 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2023-10-31T12:04:46.975 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-10-31T12:04:46.975 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.976 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.976 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2023-10-31T12:04:46.977 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.977 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:04:46.978 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.978 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.978 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2023-10-31T12:04:46.980 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:04:46.980 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:04:46.981 INFO:teuthology.orchestra.run.smithi161.stderr:12:04:46 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:05:47.042 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.043 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.043 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:05:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:05:47.140 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.141 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.141 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2023-10-31T12:05:47.157 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.157 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi161.front.sepia.ceph.com ; done 2023-10-31T12:05:47.161 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.161 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.162 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2023-10-31T12:05:47.164 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.164 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:05:47.165 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2023-10-31T12:05:47.168 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.169 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:05:47.169 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:05:47.169 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.169 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.169 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2023-10-31T12:05:47.461 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2023-10-31T12:05:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:48 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:48 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2023-10-31T12:05:49.247 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:49 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:49.247 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:49 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:49.247 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2023-10-31T12:05:50.248 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:50 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:50.248 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:50 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:50.248 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:50 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2023-10-31T12:05:51.255 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:51 - INFO - cbt - Checking Health after pool creation. 2023-10-31T12:05:56.260 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:56 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:05:56.261 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:56 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:05:56.261 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-31T12:05:56.571 INFO:teuthology.orchestra.run.smithi161.stderr:12:05:56 - INFO - cbt - Final Pool Health Check. 2023-10-31T12:06:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-31T12:06:01.875 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:01.875 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:01.876 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2023-10-31T12:06:01.878 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:01.878 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:01.878 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi161.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2023-10-31T12:06:03.654 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.655 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.655 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi161.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2023-10-31T12:06:03.754 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.755 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.755 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:06:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:06:03.825 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - INFO - cbt - Attempting to populating fio files... 2023-10-31T12:06:03.825 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.825 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.826 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-10-31T12:06:03.826 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.827 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:06:03.827 INFO:teuthology.orchestra.run.smithi161.stderr:12:06:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-10-31T12:08:29.527 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.527 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c. 2023-10-31T12:08:29.528 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - INFO - cbt - Setting OSD Read Ahead to: 4096 2023-10-31T12:08:29.528 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.528 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.528 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2023-10-31T12:08:29.529 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2023-10-31T12:08:29.529 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.530 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.530 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2023-10-31T12:08:29.544 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.545 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:29.545 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-31T12:08:30.894 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:30 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-31T12:08:31.630 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:31 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:31.632 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:31 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:31.633 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2023-10-31T12:08:31.763 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:31 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:31.763 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:31 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:31.763 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2023-10-31T12:08:57.756 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:57 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:57.757 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:57 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:57.757 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2023-10-31T12:08:58.158 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.159 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.159 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl 2023-10-31T12:08:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:08:58.163 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.163 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.163 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf 2023-10-31T12:08:58.165 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:08:58.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:08:58.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - INFO - cbt - Running rbd fio randread test. 2023-10-31T12:08:58.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2023-10-31T12:08:58.166 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.167 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.167 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 2023-10-31T12:08:58.168 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2023-10-31T12:08:58.168 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.168 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:08:58.168 INFO:teuthology.orchestra.run.smithi161.stderr:12:08:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 2023-10-31T12:09:59.242 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.243 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.243 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:09:59.264 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.264 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.264 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:09:59.294 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2023-10-31T12:09:59.296 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.296 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2023-10-31T12:09:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2023-10-31T12:09:59.586 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.624 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.624 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2023-10-31T12:09:59.640 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - get_fqdn_local()=smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.641 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi161.front.sepia.ceph.com ; done 2023-10-31T12:09:59.672 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - INFO - cbt - Convert results to json format. 2023-10-31T12:09:59.672 INFO:teuthology.orchestra.run.smithi161.stderr:12:09:59 - DEBUG - cbt - Nodes : ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.673 INFO:teuthology.orchestra.run.smithi161.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2023-10-31T12:09:59.673 INFO:teuthology.orchestra.run.smithi161.stdout:smithi161.front.sepia.ceph.com: smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.673 INFO:teuthology.orchestra.run.smithi161.stdout: 2023-10-31T12:09:59.673 INFO:teuthology.orchestra.run.smithi161.stdout:['smithi161.front.sepia.ceph.com'] 2023-10-31T12:09:59.693 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-10-31T12:09:59.703 DEBUG:teuthology.orchestra.run.smithi161:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-10-31T12:09:59.881 DEBUG:teuthology.orchestra.run.smithi161:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2023-10-31T12:09:59.936 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-10-31T12:09:59.937 INFO:tasks.cbt_performance:reading cbt preferences 2023-10-31T12:09:59.937 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-10-31T12:09:59.973 INFO:teuthology.orchestra.run.smithi161.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.974 INFO:teuthology.orchestra.run.smithi161.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi161.front.sepia.ceph.com 2023-10-31T12:09:59.975 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:09:59.975 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi161.front.sepia.ceph.com of=/dev/stdout 2023-10-31T12:10:00.032 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T12:10:00.032 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2023-10-31T12:10:00.087 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T12:10:00.087 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T12:10:00.087 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2023-10-31T12:10:00.088 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi161.front.sepia.ceph.com of=/dev/stdout 2023-10-31T12:10:00.143 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T12:10:00.144 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2023-10-31T12:10:00.197 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T12:10:00.198 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T12:10:00.210 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T12:10:00.219 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T12:10:00.219 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-31T12:10:00.228 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-31T12:10:00.345 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:10:00.337+0000 7f277eaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:10:00.346 INFO:teuthology.orchestra.run.smithi161.stderr:2023-10-31T12:10:00.345+0000 7f277eaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:10:00.541 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-31T12:10:00.541 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-31T12:10:00.542 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-31T12:10:00.542 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:10:02.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-31T12:10:06.644 INFO:tasks.ceph.osd.0:Stopped 2023-10-31T12:10:06.644 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-31T12:10:06.645 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:10:08.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-31T12:10:08.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-31T12:10:12.746 INFO:tasks.ceph.osd.1:Stopped 2023-10-31T12:10:12.747 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-31T12:10:12.747 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:10:13.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-31T12:10:13.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-31T12:10:18.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-10-31T12:10:18.849 INFO:tasks.ceph.osd.2:Stopped 2023-10-31T12:10:18.850 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-31T12:10:18.850 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-31T12:10:18.851 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:10:18.881 INFO:tasks.ceph.mgr.x:Stopped 2023-10-31T12:10:18.881 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-31T12:10:18.882 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-31T12:10:18.882 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:10:18.935 INFO:tasks.ceph.mon.a:Stopped 2023-10-31T12:10:18.935 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-31T12:10:18.936 DEBUG:teuthology.orchestra.run.smithi161:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-31T12:10:19.070 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:10:19.070 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-31T12:10:19.327 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:10:19.327 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-31T12:10:19.430 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:10:19.431 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-31T12:10:19.480 INFO:tasks.ceph:Archiving mon data... 2023-10-31T12:10:19.482 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957/data/mon.a.tgz 2023-10-31T12:10:19.482 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2023-10-31T12:10:19.501 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.nuOIMGUVdO 2023-10-31T12:10:19.501 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.nuOIMGUVdO 2023-10-31T12:10:19.638 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.nuOIMGUVdO 2023-10-31T12:10:19.746 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.nuOIMGUVdO is 205KB 2023-10-31T12:10:19.839 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.nuOIMGUVdO 2023-10-31T12:10:19.854 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-31T12:10:19.854 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-31T12:10:19.919 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-31T12:10:19.920 DEBUG:teuthology.orchestra.run.smithi161:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-31T12:10:19.996 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-31T12:10:20.000 INFO:tasks.ceph:Archiving crash dumps... 2023-10-31T12:10:20.002 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957/remote/smithi161/crash 2023-10-31T12:10:20.003 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-31T12:10:20.066 INFO:tasks.ceph:Compressing logs... 2023-10-31T12:10:20.067 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-31T12:10:20.151 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-31T12:10:20.151 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81295.log 2023-10-31T12:10:20.151 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82153.log 2023-10-31T12:10:20.156 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86086.log 2023-10-31T12:10:20.156 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81295.log: /var/log/ceph/ceph-client.admin.82153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82421.log 2023-10-31T12:10:20.156 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86582.log 2023-10-31T12:10:20.157 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82507.log 2023-10-31T12:10:20.157 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86282.log 2023-10-31T12:10:20.157 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86582.log.gz 2023-10-31T12:10:20.157 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82294.log 2023-10-31T12:10:20.157 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82507.log: /var/log/ceph/ceph-client.admin.86282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86533.log 2023-10-31T12:10:20.158 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81375.log 2023-10-31T12:10:20.158 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86250.log 2023-10-31T12:10:20.158 INFO:teuthology.orchestra.run.smithi161.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.86533.log.gz 2023-10-31T12:10:20.158 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86163.log 2023-10-31T12:10:20.159 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-31T12:10:20.159 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86581.log 2023-10-31T12:10:20.159 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86163.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81533.log 2023-10-31T12:10:20.160 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.86581.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82026.log 2023-10-31T12:10:20.160 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86581.log.gz 2023-10-31T12:10:20.160 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.81533.log: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-31T12:10:20.160 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86138.log 2023-10-31T12:10:20.160 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81335.log 2023-10-31T12:10:20.161 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.86138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82293.log 2023-10-31T12:10:20.161 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86186.log 2023-10-31T12:10:20.161 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.82293.log: /var/log/ceph/ceph-client.admin.82591.log 2023-10-31T12:10:20.161 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82420.log 2023-10-31T12:10:20.161 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82591.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-31T12:10:20.162 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86282.log.gz 2023-10-31T12:10:20.162 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81968.log 2023-10-31T12:10:20.162 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86281.log/var/log/ceph/ceph-mgr.x.log: 2023-10-31T12:10:20.167 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82110.log 2023-10-31T12:10:20.168 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82419.log 2023-10-31T12:10:20.168 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86281.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.82110.log: -- /var/log/ceph/ceph-osd.0.log 2023-10-31T12:10:20.168 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86022.log 2023-10-31T12:10:20.168 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82295.log 2023-10-31T12:10:20.169 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.86022.log: -5 --verbose -- /var/log/ceph/ceph.log 2023-10-31T12:10:20.169 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81185.log 2023-10-31T12:10:20.169 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-31T12:10:20.169 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81563.log 2023-10-31T12:10:20.171 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81185.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82549.log 2023-10-31T12:10:20.171 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86281.log.gz 2023-10-31T12:10:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86113.log 2023-10-31T12:10:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82068.log 2023-10-31T12:10:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.86113.log: -- /var/log/ceph/ceph-client.admin.82213.log 2023-10-31T12:10:20.173 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82068.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79737.log 2023-10-31T12:10:20.173 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.82213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81097.log 2023-10-31T12:10:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86209.log 2023-10-31T12:10:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.tmp-client.admin.79737.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82026.log.gz 2023-10-31T12:10:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.81097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86881.log 2023-10-31T12:10:20.175 INFO:teuthology.orchestra.run.smithi161.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82110.log.gz 2023-10-31T12:10:20.175 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86233.log 2023-10-31T12:10:20.176 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82153.log.gz 2023-10-31T12:10:20.176 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86881.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86881.log.gz 2023-10-31T12:10:20.176 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81185.log.gz 2023-10-31T12:10:20.176 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.86233.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82068.log.gz 2023-10-31T12:10:20.176 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82213.log.gz 2023-10-31T12:10:20.177 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82295.log.gz 2023-10-31T12:10:20.177 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82293.log.gz 2023-10-31T12:10:20.177 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82294.log.gz 2023-10-31T12:10:20.177 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82420.log.gz 2023-10-31T12:10:20.177 INFO:teuthology.orchestra.run.smithi161.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82421.log.gz 2023-10-31T12:10:20.178 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82419.log.gz 2023-10-31T12:10:20.178 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82507.log.gz 2023-10-31T12:10:20.178 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82549.log.gz 2023-10-31T12:10:20.178 INFO:teuthology.orchestra.run.smithi161.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82591.log.gz 2023-10-31T12:10:20.178 INFO:teuthology.orchestra.run.smithi161.stderr: 68.3% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79737.log.gz 2023-10-31T12:10:20.179 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81097.log.gz 2023-10-31T12:10:20.179 INFO:teuthology.orchestra.run.smithi161.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.86233.log.gz 2023-10-31T12:10:20.179 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86209.log.gz 2023-10-31T12:10:20.181 INFO:teuthology.orchestra.run.smithi161.stderr: 86.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-31T12:10:20.181 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81295.log.gz 2023-10-31T12:10:20.181 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81335.log.gz 2023-10-31T12:10:20.181 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81375.log.gz 2023-10-31T12:10:20.182 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81533.log.gz 2023-10-31T12:10:20.182 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81563.log.gz 2023-10-31T12:10:20.182 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81968.log.gz 2023-10-31T12:10:20.182 INFO:teuthology.orchestra.run.smithi161.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-31T12:10:20.182 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86022.log.gz 2023-10-31T12:10:20.183 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86086.log.gz 2023-10-31T12:10:20.183 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86113.log.gz 2023-10-31T12:10:20.183 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-31T12:10:20.222 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86138.log.gz 2023-10-31T12:10:20.223 INFO:teuthology.orchestra.run.smithi161.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86163.log.gz 2023-10-31T12:10:20.248 INFO:teuthology.orchestra.run.smithi161.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86186.log.gz 2023-10-31T12:10:20.248 INFO:teuthology.orchestra.run.smithi161.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.86250.log.gz 2023-10-31T12:10:20.353 INFO:teuthology.orchestra.run.smithi161.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-31T12:10:23.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-31T12:10:37.668 INFO:teuthology.orchestra.run.smithi161.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-31T12:10:37.888 INFO:teuthology.orchestra.run.smithi161.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-31T12:10:38.073 INFO:teuthology.orchestra.run.smithi161.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-31T12:10:38.075 INFO:teuthology.orchestra.run.smithi161.stderr: 2023-10-31T12:10:38.075 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m17.954s 2023-10-31T12:10:38.076 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m9.497s 2023-10-31T12:10:38.076 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.490s 2023-10-31T12:10:38.076 INFO:tasks.ceph:Archiving logs... 2023-10-31T12:10:38.077 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957/remote/smithi161/log 2023-10-31T12:10:38.078 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-31T12:10:39.610 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-31T12:10:39.621 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-31T12:10:39.622 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-31T12:10:39.685 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-31T12:10:39.696 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-31T12:10:39.696 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-31T12:10:39.711 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2023-10-31T12:10:39.769 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-31T12:10:39.769 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2023-10-31T12:10:39.770 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +1419us[+1415us] +/- 85ms 2023-10-31T12:10:39.770 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 +430ns[-3436ns] +/- 69ms 2023-10-31T12:10:39.770 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -1288us[-1292us] +/- 76ms 2023-10-31T12:10:39.770 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:10:39.777 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-31T12:10:39.789 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-31T12:10:39.789 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-31T12:10:39.800 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-31T12:10:39.972 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-31T12:10:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T12:10:40.006 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-31T12:10:40.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-31T12:10:40.032 INFO:teuthology.task.internal:Duration was 948.495154 seconds 2023-10-31T12:10:40.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-31T12:10:40.043 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-31T12:10:40.043 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-31T12:10:40.129 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-31T12:10:40.237 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-31T12:10:40.238 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2023-10-31T12:10:40.238 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-31T12:10:40.257 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-31T12:10:40.257 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-31T12:10:40.325 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-31T12:10:40.326 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-31T12:10:40.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-31T12:10:40.495 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-31T12:10:40.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-31T12:10:40.533 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-31T12:10:40.592 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-31T12:10:40.637 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2023-10-31T12:10:40.656 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-31T12:10:40.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T12:10:40.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-31T12:10:40.684 INFO:teuthology.task.internal:Transferring archived files... 2023-10-31T12:10:40.685 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441957/remote/smithi161 2023-10-31T12:10:40.687 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-31T12:10:40.785 INFO:teuthology.task.internal:Removing archive directory... 2023-10-31T12:10:40.786 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-31T12:10:40.818 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-31T12:10:40.835 INFO:teuthology.task.internal:Not uploading archives. 2023-10-31T12:10:40.835 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-31T12:10:40.849 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-31T12:10:40.849 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-31T12:10:40.873 INFO:teuthology.orchestra.run.smithi161.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 31 12:10 /home/ubuntu/cephtest 2023-10-31T12:10:40.874 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-31T12:10:40.887 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-31T12:10:40.922 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-wip-matanb-crimson-do_osd_ops_execute-v3&var-job_id_selected=7441957 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} duration: 948.4951536655426 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-10-31T12:10:40.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-31T12:10:40.999 INFO:teuthology.run:pass