2024-04-01T02:32:35.348 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-01T02:32:35.348 DEBUG:teuthology.run:Teuthology command: teuthology --name yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815 --verbose -- /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815/orig.config.yaml 2024-04-01T02:32:35.397 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T02:32:35.490 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815 branch: ci-yingxin-crimson-load-obc-3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7633815' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-3 env: CRIMSON_COMPAT: '1' sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 targets: smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC7tFyiPlmPfRK7Fy3xFqDwO/tHWKJVaLJwXnBtAa8JHt21rV/oXjOPSkkgcrc2QRaO6K0jYmC3ZC+KY3lzUNI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-01_02:22:08 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-01T02:32:35.490 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa; will attempt to use it 2024-04-01T02:32:35.491 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks 2024-04-01T02:32:35.491 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-01T02:32:35.493 INFO:teuthology.task.internal:Checking packages... 2024-04-01T02:32:35.518 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '785fc5e8d093374f751c7da3e51e9b56b3718d17' 2024-04-01T02:32:35.518 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-01T02:32:35.518 INFO:teuthology.packaging:ref: None 2024-04-01T02:32:35.518 INFO:teuthology.packaging:tag: None 2024-04-01T02:32:35.518 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-3 2024-04-01T02:32:35.518 INFO:teuthology.packaging:sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-04-01T02:32:35.518 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-3 2024-04-01T02:32:35.668 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2473.g785fc5e8 2024-04-01T02:32:35.669 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-01T02:32:35.676 INFO:teuthology.task.internal:no buildpackages task found 2024-04-01T02:32:35.676 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-01T02:32:35.719 INFO:teuthology.task.internal:Saving configuration 2024-04-01T02:32:35.736 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-01T02:32:35.744 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T02:32:35.770 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 02:27:47.116214', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC7tFyiPlmPfRK7Fy3xFqDwO/tHWKJVaLJwXnBtAa8JHt21rV/oXjOPSkkgcrc2QRaO6K0jYmC3ZC+KY3lzUNI='} 2024-04-01T02:32:35.770 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-01T02:32:35.776 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-01T02:32:35.776 INFO:teuthology.run_tasks:Running task console_log... 2024-04-01T02:32:35.837 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a4483df70>, signals=[15]) 2024-04-01T02:32:35.837 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-01T02:32:35.843 INFO:teuthology.task.internal:Opening connections... 2024-04-01T02:32:35.843 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:32:35.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T02:32:35.935 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-01T02:32:35.942 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-04-01T02:32:36.006 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-04-01T02:32:36.006 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-04-01T02:32:36.062 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T02:32:36.063 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-04-01T02:32:36.063 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T02:32:36.063 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T02:32:36.063 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T02:32:36.064 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-04-01T02:32:36.096 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-01T02:32:36.105 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-01T02:32:36.169 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-01T02:32:36.169 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-04-01T02:32:36.185 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-01T02:32:36.193 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-01T02:32:36.193 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-04-01T02:32:36.263 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-01T02:32:36.295 INFO:teuthology.run_tasks:Running task kernel... 2024-04-01T02:32:36.311 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-01T02:32:36.311 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-01T02:32:36.312 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T02:32:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:32:36.328 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-01T02:32:36.384 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-361.el9.x86_64 2024-04-01T02:32:36.384 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-361.el9.x86_64 2024-04-01T02:32:36.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-04-01T02:32:38.592 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-01T02:32:40.201 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-04-01T02:32:41.283 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-01T02:32:43.975 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-01T02:32:48.042 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-04-01T02:32:50.459 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-01T02:32:54.986 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 21 MB 00:04 2024-04-01T02:32:59.877 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-01T02:33:02.167 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T02:33:02.167 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T02:33:02.291 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:33:02.296 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:33:02.296 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:33:02.296 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:33:02.296 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2024-04-01T02:33:02.297 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:02.298 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 76 M 2024-04-01T02:33:02.298 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 123 M 2024-04-01T02:33:02.298 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:33:02.445 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-01T02:33:02.953 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-04-01T02:33:03.562 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-04-01T02:33:03.879 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-01T02:33:03.882 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:33:03.882 INFO:teuthology.orchestra.run.smithi187.stdout:Total 48 MB/s | 76 MB 00:01 2024-04-01T02:33:03.882 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:33:04.261 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:33:04.261 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:33:05.121 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:33:05.121 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:33:07.519 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:33:08.326 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T02:33:08.420 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T02:33:11.145 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T02:33:11.280 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T02:33:16.016 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T02:33:16.128 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T02:33:20.859 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T02:33:50.355 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T02:33:50.355 INFO:teuthology.orchestra.run.smithi187.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T02:33:50.355 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:50.398 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T02:33:50.741 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T02:33:50.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T02:33:50.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T02:33:50.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:51.131 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:33:51.463 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T02:33:51.571 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T02:33:51.571 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T02:33:51.571 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T02:33:51.571 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T02:33:51.571 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T02:33:51.572 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-01T02:33:51.572 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T02:33:51.572 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T02:33:51.651 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T02:33:51.652 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T02:33:51.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.06 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 68.el9 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Group : Unspecified 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 8264254 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2024-04-01T02:33:51.690 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : builder@centos.org 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:33:51.691 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T02:33:51.692 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T02:33:51.692 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 5.14.0-434.el9.x86_64 2024-04-01T02:33:51.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T02:33:52.288 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-04-01T02:33:53.577 INFO:teuthology.orchestra.run.smithi187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T02:33:53.630 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-04-01T02:33:53.632 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-01T02:33:53.649 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.D2FskXtoZX 2024-04-01T02:33:53.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.D2FskXtoZX 2024-04-01T02:33:53.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.D2FskXtoZX 2024-04-01T02:33:53.857 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.D2FskXtoZX is 6KB 2024-04-01T02:33:53.906 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.D2FskXtoZX 2024-04-01T02:33:53.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T02:33:53.986 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T02:33:53.987 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T02:33:53.987 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T02:33:53.987 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T02:33:53.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T02:33:54.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-04-01T02:34:24.146 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-01T02:34:24.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-01T02:34:24.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T02:34:42.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-01T02:34:51.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-01T02:34:51.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T02:34:54.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-01T02:35:06.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-01T02:35:06.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T02:35:38.410 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-01T02:35:38.841 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-01T02:35:38.841 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-01T02:35:38.842 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-01T02:35:38.856 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-434.el9.x86_64 2024-04-01T02:35:38.856 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T02:35:38.856 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T02:35:38.856 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T02:35:39.857 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-01T02:35:39.857 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T02:35:39.916 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-04-01T02:35:39.969 DEBUG:teuthology.parallel:result is None 2024-04-01T02:35:39.970 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-01T02:35:39.979 INFO:teuthology.task.internal:Creating test directory... 2024-04-01T02:35:39.979 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T02:35:39.994 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-01T02:35:40.017 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-01T02:35:40.042 INFO:teuthology.task.internal:Creating archive directory... 2024-04-01T02:35:40.043 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T02:35:40.095 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-01T02:35:40.104 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-01T02:35:40.104 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T02:35:40.276 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T02:35:40.289 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T02:35:40.291 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-01T02:35:40.299 INFO:teuthology.task.internal:Configuring sudo... 2024-04-01T02:35:40.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T02:35:40.327 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-01T02:35:40.341 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-01T02:35:40.341 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T02:35:40.382 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T02:35:40.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T02:35:40.538 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T02:35:40.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T02:35:40.656 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:35:40.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T02:35:40.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-04-01T02:35:40.805 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T02:35:41.353 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-01T02:35:41.363 INFO:teuthology.task.internal:Starting timer... 2024-04-01T02:35:41.363 INFO:teuthology.run_tasks:Running task pcp... 2024-04-01T02:35:41.403 INFO:teuthology.run_tasks:Running task selinux... 2024-04-01T02:35:41.424 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-01T02:35:41.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-04-01T02:35:41.482 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-04-01T02:35:41.484 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-01T02:35:41.484 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-04-01T02:35:41.506 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-04-01T02:35:41.507 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-04-01T02:35:41.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T02:35:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:35:41.575 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-01T02:35:41.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-04-01T02:35:41.649 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-01T02:35:41.663 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-01T02:35:41.668 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-01T02:35:41.786 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-01T02:35:41.802 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-01T02:35:41.817 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-01T02:40:04.730 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-04-01T02:40:04.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-01T02:40:04.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T02:40:04.824 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-01T02:40:04.901 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-01T02:40:04.902 INFO:teuthology.run_tasks:Running task clock... 2024-04-01T02:40:04.913 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-01T02:40:04.913 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T02:40:04.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T02:40:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T02:40:04.992 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T02:40:05.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-04-01T02:40:05.035 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-04-01T02:40:05.051 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T02:40:05.068 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T02:40:05.119 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-01T02:40:05.122 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T02:40:05.122 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-01T02:40:05.122 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:40:05.122 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:40:05.123 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:40:05.123 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:40:05.124 INFO:teuthology.run_tasks:Running task install... 2024-04-01T02:40:05.132 DEBUG:teuthology.task.install:project ceph 2024-04-01T02:40:05.132 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}} 2024-04-01T02:40:05.132 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'} 2024-04-01T02:40:05.132 INFO:teuthology.task.install:Using flavor: crimson 2024-04-01T02:40:05.137 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-01T02:40:05.138 INFO:teuthology.task.install:extra packages: [] 2024-04-01T02:40:05.138 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'tag': None, 'wait_for_package': False} 2024-04-01T02:40:05.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-04-01T02:40:05.371 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/ 2024-04-01T02:40:05.371 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2473.g785fc5e8 2024-04-01T02:40:05.485 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T02:40:05.485 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:40:05.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T02:40:05.514 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-01T02:40:05.514 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/785fc5e8d093374f751c7da3e51e9b56b3718d17/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T02:40:05.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T02:40:05.663 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T02:40:05.730 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-04-01T02:40:05.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-04-01T02:40:05.971 INFO:teuthology.orchestra.run.smithi187.stdout:74 files removed 2024-04-01T02:40:05.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2024-04-01T02:40:06.764 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 186 kB/s | 87 kB 00:00 2024-04-01T02:40:07.090 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-04-01T02:40:07.507 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 5.4 kB/s | 2.0 kB 00:00 2024-04-01T02:40:07.893 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-04-01T02:40:09.410 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-04-01T02:40:11.759 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:02 2024-04-01T02:40:13.593 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-01T02:40:17.569 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-04-01T02:40:19.170 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-01T02:40:20.440 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-01T02:40:25.346 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-01T02:40:27.968 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 9.6 M 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-04-01T02:40:27.971 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.7 M 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.9 M 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 19 M 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 25 k 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 796 k 2024-04-01T02:40:27.972 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 547 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 45 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 134 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 159 k 2024-04-01T02:40:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 315 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 293 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 99 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-04-01T02:40:27.974 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:Install 30 Packages 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 58 M 2024-04-01T02:40:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:40:29.114 INFO:teuthology.orchestra.run.smithi187.stdout:(1/32): ceph-base-19.0.0-2473.g785fc5e8.el9.x86 5.4 MB/s | 5.9 MB 00:01 2024-04-01T02:40:29.172 INFO:teuthology.orchestra.run.smithi187.stdout:(2/32): ceph-selinux-19.0.0-2473.g785fc5e8.el9. 433 kB/s | 25 kB 00:00 2024-04-01T02:40:29.357 INFO:teuthology.orchestra.run.smithi187.stdout:(3/32): libcephfs2-19.0.0-2473.g785fc5e8.el9.x8 4.2 MB/s | 796 kB 00:00 2024-04-01T02:40:29.599 INFO:teuthology.orchestra.run.smithi187.stdout:(4/32): ceph-radosgw-19.0.0-2473.g785fc5e8.el9. 6.1 MB/s | 9.6 MB 00:01 2024-04-01T02:40:29.624 INFO:teuthology.orchestra.run.smithi187.stdout:(5/32): libradosstriper1-19.0.0-2473.g785fc5e8. 2.0 MB/s | 547 kB 00:00 2024-04-01T02:40:29.717 INFO:teuthology.orchestra.run.smithi187.stdout:(6/32): python3-ceph-argparse-19.0.0-2473.g785f 491 kB/s | 45 kB 00:00 2024-04-01T02:40:29.825 INFO:teuthology.orchestra.run.smithi187.stdout:(7/32): python3-ceph-common-19.0.0-2473.g785fc5 1.2 MB/s | 134 kB 00:00 2024-04-01T02:40:29.926 INFO:teuthology.orchestra.run.smithi187.stdout:(8/32): python3-cephfs-19.0.0-2473.g785fc5e8.el 1.6 MB/s | 159 kB 00:00 2024-04-01T02:40:30.026 INFO:teuthology.orchestra.run.smithi187.stdout:(9/32): python3-rados-19.0.0-2473.g785fc5e8.el9 3.1 MB/s | 315 kB 00:00 2024-04-01T02:40:30.226 INFO:teuthology.orchestra.run.smithi187.stdout:(10/32): ceph-common-19.0.0-2473.g785fc5e8.el9. 8.6 MB/s | 19 MB 00:02 2024-04-01T02:40:30.251 INFO:teuthology.orchestra.run.smithi187.stdout:(11/32): python3-rbd-19.0.0-2473.g785fc5e8.el9. 1.3 MB/s | 293 kB 00:00 2024-04-01T02:40:30.277 INFO:teuthology.orchestra.run.smithi187.stdout:(12/32): python3-rgw-19.0.0-2473.g785fc5e8.el9. 1.9 MB/s | 99 kB 00:00 2024-04-01T02:40:30.352 INFO:teuthology.orchestra.run.smithi187.stdout:(13/32): librgw2-19.0.0-2473.g785fc5e8.el9.x86_ 6.7 MB/s | 5.0 MB 00:00 2024-04-01T02:40:30.477 INFO:teuthology.orchestra.run.smithi187.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 469 kB/s | 106 kB 00:00 2024-04-01T02:40:30.502 INFO:teuthology.orchestra.run.smithi187.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 706 kB/s | 159 kB 00:00 2024-04-01T02:40:30.519 INFO:teuthology.orchestra.run.smithi187.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 373 kB/s | 44 kB 00:00 2024-04-01T02:40:30.628 INFO:teuthology.orchestra.run.smithi187.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-04-01T02:40:30.653 INFO:teuthology.orchestra.run.smithi187.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-04-01T02:40:30.678 INFO:teuthology.orchestra.run.smithi187.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 259 kB/s | 41 kB 00:00 2024-04-01T02:40:30.703 INFO:teuthology.orchestra.run.smithi187.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-04-01T02:40:30.729 INFO:teuthology.orchestra.run.smithi187.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 929 kB/s | 46 kB 00:00 2024-04-01T02:40:30.754 INFO:teuthology.orchestra.run.smithi187.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 463 kB/s | 35 kB 00:00 2024-04-01T02:40:30.838 INFO:teuthology.orchestra.run.smithi187.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-04-01T02:40:30.880 INFO:teuthology.orchestra.run.smithi187.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-04-01T02:40:30.905 INFO:teuthology.orchestra.run.smithi187.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 726 kB/s | 49 kB 00:00 2024-04-01T02:40:30.930 INFO:teuthology.orchestra.run.smithi187.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-01T02:40:31.014 INFO:teuthology.orchestra.run.smithi187.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-01T02:40:31.039 INFO:teuthology.orchestra.run.smithi187.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-01T02:40:31.081 INFO:teuthology.orchestra.run.smithi187.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-01T02:40:31.106 INFO:teuthology.orchestra.run.smithi187.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-04-01T02:40:32.581 INFO:teuthology.orchestra.run.smithi187.stdout:(31/32): librados2-19.0.0-2473.g785fc5e8.el9.x8 2.2 MB/s | 3.4 MB 00:01 2024-04-01T02:40:33.248 INFO:teuthology.orchestra.run.smithi187.stdout:(32/32): librbd1-19.0.0-2473.g785fc5e8.el9.x86_ 1.7 MB/s | 3.7 MB 00:02 2024-04-01T02:40:33.253 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:40:33.253 INFO:teuthology.orchestra.run.smithi187.stdout:Total 11 MB/s | 58 MB 00:05 2024-04-01T02:40:33.254 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:40:33.292 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:40:33.292 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:40:33.770 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:40:33.770 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:40:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:40:35.739 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-01T02:40:35.792 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-04-01T02:40:35.884 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-04-01T02:40:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-04-01T02:40:36.002 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-04-01T02:40:36.064 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-04-01T02:40:36.111 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-01T02:40:36.140 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-01T02:40:36.173 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-01T02:40:36.188 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-01T02:40:36.204 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-01T02:40:36.383 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-01T02:40:36.412 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-01T02:40:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-01T02:40:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-04-01T02:40:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-04-01T02:40:36.885 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 12/34 2024-04-01T02:40:36.944 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-04-01T02:40:36.973 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 14/34 2024-04-01T02:40:37.062 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-01T02:40:37.103 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-04-01T02:40:37.200 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-04-01T02:40:37.235 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-01T02:40:37.355 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-01T02:40:37.395 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-01T02:40:37.435 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-01T02:40:37.468 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-01T02:40:37.497 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-01T02:40:37.528 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-01T02:40:37.567 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-01T02:40:38.108 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-01T02:40:38.135 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-04-01T02:40:38.191 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-04-01T02:40:38.265 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 27/34 2024-04-01T02:40:38.304 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 28/34 2024-04-01T02:40:40.672 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-04-01T02:40:40.708 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-04-01T02:40:41.390 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-04-01T02:40:41.428 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-04-01T02:40:41.504 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-04-01T02:40:41.504 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T02:40:41.504 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T02:40:41.504 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:41.538 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-04-01T02:40:51.621 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-04-01T02:40:51.621 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-04-01T02:40:51.621 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-04-01T02:40:51.621 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-04-01T02:40:51.622 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-04-01T02:40:51.622 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-04-01T02:40:51.622 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2024-04-01T02:40:51.622 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-04-01T02:40:51.622 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:52.648 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-04-01T02:40:52.673 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-04-01T02:40:52.674 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:40:52.674 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T02:40:52.674 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T02:40:52.674 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T02:40:52.674 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:52.675 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T02:40:52.754 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T02:40:52.754 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 6/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 7/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 8/34 2024-04-01T02:40:53.964 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 9/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 12/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-01T02:40:53.965 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-01T02:40:53.966 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/34 2024-04-01T02:40:54.978 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T02:40:54.979 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.980 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:54.981 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:40:55.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2024-04-01T02:40:55.796 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:30 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:40:55.948 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 87 M 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 174 k 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-01T02:40:55.949 INFO:teuthology.orchestra.run.smithi187.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout:Install 11 Packages 2024-04-01T02:40:55.950 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:40:55.951 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 90 M 2024-04-01T02:40:55.951 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 819 M 2024-04-01T02:40:55.951 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:40:56.258 INFO:teuthology.orchestra.run.smithi187.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-04-01T02:40:56.366 INFO:teuthology.orchestra.run.smithi187.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2024-04-01T02:40:56.408 INFO:teuthology.orchestra.run.smithi187.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-01T02:40:56.442 INFO:teuthology.orchestra.run.smithi187.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-01T02:40:56.476 INFO:teuthology.orchestra.run.smithi187.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 90 kB 00:00 2024-04-01T02:40:56.684 INFO:teuthology.orchestra.run.smithi187.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 899 kB/s | 187 kB 00:00 2024-04-01T02:40:56.768 INFO:teuthology.orchestra.run.smithi187.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 5.5 MB/s | 469 kB 00:00 2024-04-01T02:40:56.852 INFO:teuthology.orchestra.run.smithi187.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-04-01T02:40:56.894 INFO:teuthology.orchestra.run.smithi187.stdout:(9/11): libcephsqlite-19.0.0-2473.g785fc5e8.el9 219 kB/s | 174 kB 00:00 2024-04-01T02:40:57.011 INFO:teuthology.orchestra.run.smithi187.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 768 kB/s | 122 kB 00:00 2024-04-01T02:41:12.668 INFO:teuthology.orchestra.run.smithi187.stdout:(11/11): ceph-test-19.0.0-2473.g785fc5e8.el9.x8 5.3 MB/s | 87 MB 00:16 2024-04-01T02:41:12.671 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:41:12.671 INFO:teuthology.orchestra.run.smithi187.stdout:Total 5.4 MB/s | 90 MB 00:16 2024-04-01T02:41:12.712 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:41:12.735 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:41:12.736 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:41:13.377 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:41:13.377 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:41:14.371 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:41:14.438 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-01T02:41:14.468 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-01T02:41:14.494 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-01T02:41:14.565 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-01T02:41:14.593 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-01T02:41:14.637 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-01T02:41:14.682 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-01T02:41:14.714 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-01T02:41:14.753 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-01T02:41:14.798 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-01T02:41:14.824 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-04-01T02:41:26.416 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-04-01T02:41:26.489 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-04-01T02:41:32.860 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-04-01T02:41:32.860 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-01T02:41:32.861 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.6-16.el9.x86_64 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:41:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:33.270 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:41:33.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2024-04-01T02:41:34.121 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:09 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:41:34.274 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:41:34.277 INFO:teuthology.orchestra.run.smithi187.stdout:========================================================================================= 2024-04-01T02:41:34.277 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:41:34.277 INFO:teuthology.orchestra.run.smithi187.stdout:========================================================================================= 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 6.5 k 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 2.2 M 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 1.6 M 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 249 k 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 71 M 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T02:41:34.278 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T02:41:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T02:41:34.280 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout:========================================================================================= 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout:Install 36 Packages 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 84 M 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 921 M 2024-04-01T02:41:34.281 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:41:35.207 INFO:teuthology.orchestra.run.smithi187.stdout:(1/36): ceph-19.0.0-2473.g785fc5e8.el9.x86_64.r 21 kB/s | 6.5 kB 00:00 2024-04-01T02:41:35.708 INFO:teuthology.orchestra.run.smithi187.stdout:(2/36): ceph-mon-19.0.0-2473.g785fc5e8.el9.x86_ 10 MB/s | 5.0 MB 00:00 2024-04-01T02:41:37.858 INFO:teuthology.orchestra.run.smithi187.stdout:(3/36): ceph-mgr-19.0.0-2473.g785fc5e8.el9.x86_ 537 kB/s | 1.6 MB 00:02 2024-04-01T02:41:38.233 INFO:teuthology.orchestra.run.smithi187.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2473.g785f 663 kB/s | 249 kB 00:00 2024-04-01T02:41:38.358 INFO:teuthology.orchestra.run.smithi187.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-04-01T02:41:38.400 INFO:teuthology.orchestra.run.smithi187.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-01T02:41:38.434 INFO:teuthology.orchestra.run.smithi187.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-04-01T02:41:38.484 INFO:teuthology.orchestra.run.smithi187.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-04-01T02:41:38.543 INFO:teuthology.orchestra.run.smithi187.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 662 kB/s | 39 kB 00:00 2024-04-01T02:41:38.610 INFO:teuthology.orchestra.run.smithi187.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-04-01T02:41:38.719 INFO:teuthology.orchestra.run.smithi187.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-04-01T02:41:38.785 INFO:teuthology.orchestra.run.smithi187.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-04-01T02:41:38.852 INFO:teuthology.orchestra.run.smithi187.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-04-01T02:41:38.911 INFO:teuthology.orchestra.run.smithi187.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-04-01T02:41:39.311 INFO:teuthology.orchestra.run.smithi187.stdout:(15/36): ceph-mds-19.0.0-2473.g785fc5e8.el9.x86 505 kB/s | 2.2 MB 00:04 2024-04-01T02:41:39.405 INFO:teuthology.orchestra.run.smithi187.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 238 kB/s | 22 kB 00:00 2024-04-01T02:41:40.372 INFO:teuthology.orchestra.run.smithi187.stdout:(17/36): ceph-osd-19.0.0-2473.g785fc5e8.el9.x86 15 MB/s | 71 MB 00:04 2024-04-01T02:41:40.564 INFO:teuthology.orchestra.run.smithi187.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 76 kB/s | 125 kB 00:01 2024-04-01T02:41:40.714 INFO:teuthology.orchestra.run.smithi187.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 115 kB/s | 151 kB 00:01 2024-04-01T02:41:40.906 INFO:teuthology.orchestra.run.smithi187.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-04-01T02:41:40.939 INFO:teuthology.orchestra.run.smithi187.stdout:(21/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 77 kB/s | 43 kB 00:00 2024-04-01T02:41:41.007 INFO:teuthology.orchestra.run.smithi187.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-01T02:41:41.040 INFO:teuthology.orchestra.run.smithi187.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-04-01T02:41:41.107 INFO:teuthology.orchestra.run.smithi187.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-04-01T02:41:41.141 INFO:teuthology.orchestra.run.smithi187.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-04-01T02:41:41.208 INFO:teuthology.orchestra.run.smithi187.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-04-01T02:41:41.249 INFO:teuthology.orchestra.run.smithi187.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-04-01T02:41:41.275 INFO:teuthology.orchestra.run.smithi187.stdout:(28/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 242 kB/s | 172 kB 00:00 2024-04-01T02:41:41.308 INFO:teuthology.orchestra.run.smithi187.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 786 kB/s | 79 kB 00:00 2024-04-01T02:41:41.375 INFO:teuthology.orchestra.run.smithi187.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-04-01T02:41:41.409 INFO:teuthology.orchestra.run.smithi187.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 897 kB/s | 90 kB 00:00 2024-04-01T02:41:41.476 INFO:teuthology.orchestra.run.smithi187.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-04-01T02:41:41.509 INFO:teuthology.orchestra.run.smithi187.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-01T02:41:41.535 INFO:teuthology.orchestra.run.smithi187.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 956 kB/s | 272 kB 00:00 2024-04-01T02:41:41.618 INFO:teuthology.orchestra.run.smithi187.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-04-01T02:41:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-04-01T02:41:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:41:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Total 11 MB/s | 84 MB 00:07 2024-04-01T02:41:41.691 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:41:41.734 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:41:41.734 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:41:42.468 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:41:42.469 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:41:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:41:44.129 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-01T02:41:44.172 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-01T02:41:44.206 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-01T02:41:44.277 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-01T02:41:44.322 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-01T02:41:44.355 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-01T02:41:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-01T02:41:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-01T02:41:44.566 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-01T02:41:44.638 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-01T02:41:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-01T02:41:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-01T02:41:44.787 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-01T02:41:44.843 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-01T02:41:45.038 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-01T02:41:45.084 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-01T02:41:45.115 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-01T02:41:45.149 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-01T02:41:45.179 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-01T02:41:45.256 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-01T02:41:45.289 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T02:41:45.322 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-01T02:41:45.363 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-01T02:41:45.398 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-01T02:41:45.441 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-01T02:41:45.531 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-01T02:41:45.571 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-01T02:41:45.643 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-01T02:41:45.782 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-01T02:41:45.907 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T02:41:46.101 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 31/36 2024-04-01T02:41:46.126 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-04-01T02:41:46.151 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-04-01T02:41:46.152 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:41:46.152 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T02:41:46.152 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T02:41:46.152 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T02:41:46.152 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:56.992 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-04-01T02:41:57.019 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-04-01T02:41:57.019 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:41:57.019 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T02:41:57.019 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T02:41:57.020 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T02:41:57.020 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:57.560 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-04-01T02:41:57.585 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-04-01T02:41:57.585 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:41:57.585 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T02:41:57.585 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T02:41:57.585 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T02:41:57.586 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:57.828 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-04-01T02:41:57.853 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-04-01T02:41:57.853 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:41:57.853 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T02:41:57.853 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T02:41:57.853 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T02:41:57.853 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:41:57.919 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-04-01T02:42:04.748 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 6/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-01T02:42:04.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-01T02:42:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-01T02:42:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:05.662 INFO:teuthology.orchestra.run.smithi187.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T02:42:05.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T02:42:05.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T02:42:05.665 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:05.665 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:05.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2024-04-01T02:42:06.486 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:41 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:06.585 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:06.638 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:06.639 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:06.639 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:06.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2024-04-01T02:42:07.254 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:42 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:07.404 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:07.405 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:07.405 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:07.405 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:07.405 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 762 k 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 762 k 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 768 k 2024-04-01T02:42:07.406 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:07.693 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-19.0.0-2473.g785fc5e8.el9.noarch.rpm 2.6 MB/s | 762 kB 00:00 2024-04-01T02:42:07.694 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:07.694 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-04-01T02:42:07.694 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:07.699 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:07.699 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:07.705 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:07.705 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:07.782 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:08.220 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-04-01T02:42:08.306 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-04-01T02:42:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-04-01T02:42:08.997 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-04-01T02:42:08.997 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:08.997 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:08.997 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:08.998 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:09.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2024-04-01T02:42:09.758 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:44 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:09.915 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:09.915 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:09.915 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 157 k 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 157 k 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 640 k 2024-04-01T02:42:09.916 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:10.341 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-19.0.0-2473.g785fc5 370 kB/s | 157 kB 00:00 2024-04-01T02:42:10.341 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:10.341 INFO:teuthology.orchestra.run.smithi187.stdout:Total 369 kB/s | 157 kB 00:00 2024-04-01T02:42:10.342 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:10.351 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:10.351 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:10.404 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:10.405 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:10.608 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-04-01T02:42:10.632 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-04-01T02:42:10.632 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:42:10.632 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T02:42:10.632 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:11.612 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-04-01T02:42:11.612 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:11.612 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:11.612 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:11.612 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:11.612 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:11.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2024-04-01T02:42:12.437 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:47 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:12.538 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:12.590 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:12.591 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:12.592 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:12.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-dashboard 2024-04-01T02:42:13.216 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:48 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:13.367 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================== 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================== 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 3.5 M 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 34 k 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 16 k 2024-04-01T02:42:13.368 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================== 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout:Install 11 Packages 2024-04-01T02:42:13.369 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:13.370 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 7.6 M 2024-04-01T02:42:13.370 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 100 M 2024-04-01T02:42:13.370 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:13.736 INFO:teuthology.orchestra.run.smithi187.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2473.g78 176 kB/s | 34 kB 00:00 2024-04-01T02:42:13.836 INFO:teuthology.orchestra.run.smithi187.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2473.g785 56 kB/s | 16 kB 00:00 2024-04-01T02:42:13.887 INFO:teuthology.orchestra.run.smithi187.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-04-01T02:42:14.045 INFO:teuthology.orchestra.run.smithi187.stdout:(4/11): ceph-mgr-dashboard-19.0.0-2473.g785fc5e 7.1 MB/s | 3.5 MB 00:00 2024-04-01T02:42:14.262 INFO:teuthology.orchestra.run.smithi187.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-04-01T02:42:14.279 INFO:teuthology.orchestra.run.smithi187.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-04-01T02:42:14.396 INFO:teuthology.orchestra.run.smithi187.stdout:(7/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-04-01T02:42:14.471 INFO:teuthology.orchestra.run.smithi187.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 416 kB/s | 31 kB 00:00 2024-04-01T02:42:14.505 INFO:teuthology.orchestra.run.smithi187.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 640 kB/s | 144 kB 00:00 2024-04-01T02:42:14.547 INFO:teuthology.orchestra.run.smithi187.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-04-01T02:42:14.905 INFO:teuthology.orchestra.run.smithi187.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.2 MB/s | 2.0 MB 00:00 2024-04-01T02:42:14.910 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:14.910 INFO:teuthology.orchestra.run.smithi187.stdout:Total 4.9 MB/s | 7.6 MB 00:01 2024-04-01T02:42:14.928 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:14.944 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:14.944 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:15.118 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:15.118 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:15.608 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-01T02:42:15.725 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-01T02:42:15.835 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-01T02:42:15.958 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-01T02:42:16.011 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-01T02:42:16.093 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-01T02:42:16.138 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-01T02:42:16.167 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-01T02:42:16.199 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 9/11 2024-04-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 10/11 2024-04-01T02:42:17.291 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-04-01T02:42:18.591 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 1/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 2/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 3/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-01T02:42:18.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-01T02:42:18.954 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-01T02:42:18.954 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:18.954 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:18.954 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T02:42:18.954 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:18.954 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:18.955 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:19.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-01T02:42:19.772 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:54 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:19.923 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:19.924 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================ 2024-04-01T02:42:19.924 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================ 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 7.4 M 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T02:42:19.925 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================ 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout:Install 12 Packages 2024-04-01T02:42:19.926 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:19.927 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 37 M 2024-04-01T02:42:19.927 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 209 M 2024-04-01T02:42:19.927 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:20.387 INFO:teuthology.orchestra.run.smithi187.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 214 kB/s | 32 kB 00:00 2024-04-01T02:42:20.412 INFO:teuthology.orchestra.run.smithi187.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 721 kB/s | 17 kB 00:00 2024-04-01T02:42:20.454 INFO:teuthology.orchestra.run.smithi187.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 828 kB/s | 34 kB 00:00 2024-04-01T02:42:20.513 INFO:teuthology.orchestra.run.smithi187.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-01T02:42:20.730 INFO:teuthology.orchestra.run.smithi187.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-04-01T02:42:20.797 INFO:teuthology.orchestra.run.smithi187.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-01T02:42:20.822 INFO:teuthology.orchestra.run.smithi187.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-04-01T02:42:21.139 INFO:teuthology.orchestra.run.smithi187.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-01T02:42:21.331 INFO:teuthology.orchestra.run.smithi187.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-04-01T02:42:21.381 INFO:teuthology.orchestra.run.smithi187.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-04-01T02:42:21.573 INFO:teuthology.orchestra.run.smithi187.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-04-01T02:42:21.990 INFO:teuthology.orchestra.run.smithi187.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 4.2 MB/s | 7.4 MB 00:01 2024-04-01T02:42:21.993 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:21.993 INFO:teuthology.orchestra.run.smithi187.stdout:Total 18 MB/s | 37 MB 00:02 2024-04-01T02:42:22.033 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:22.058 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:22.058 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:22.420 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:22.421 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:23.126 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:23.205 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-01T02:42:23.244 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-01T02:42:23.338 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-01T02:42:23.372 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-01T02:42:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T02:42:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T02:42:23.653 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-01T02:42:24.304 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-01T02:42:24.417 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-01T02:42:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-01T02:42:26.683 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-01T02:42:26.720 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 1/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-01T02:42:29.302 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-01T02:42:29.303 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-01T02:42:29.705 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T02:42:29.706 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T02:42:29.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T02:42:29.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T02:42:29.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T02:42:29.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:29.707 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:30.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-rook 2024-04-01T02:42:30.672 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:05 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:30.825 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:30.826 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2024-04-01T02:42:30.826 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 49 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T02:42:30.827 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.0 M 2024-04-01T02:42:30.828 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 26 M 2024-04-01T02:42:30.829 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:31.021 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 24 kB 00:00 2024-04-01T02:42:31.046 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2024-04-01T02:42:31.105 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-04-01T02:42:31.130 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-04-01T02:42:31.155 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-01T02:42:31.180 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-04-01T02:42:31.205 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): ceph-mgr-rook-19.0.0-2473.g785fc5e8.el9 164 kB/s | 49 kB 00:00 2024-04-01T02:42:31.389 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 138 kB/s | 32 kB 00:00 2024-04-01T02:42:31.414 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-04-01T02:42:31.556 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-04-01T02:42:31.582 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 539 kB/s | 203 kB 00:00 2024-04-01T02:42:31.665 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-04-01T02:42:31.682 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 715 kB/s | 90 kB 00:00 2024-04-01T02:42:31.683 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:31.684 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-04-01T02:42:31.684 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:31.697 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:31.697 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:31.788 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:31.788 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:32.296 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:32.460 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-01T02:42:32.532 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-01T02:42:32.596 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-01T02:42:32.634 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-01T02:42:32.676 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-01T02:42:32.768 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-01T02:42:32.868 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-01T02:42:32.927 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-01T02:42:33.647 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-01T02:42:33.691 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-01T02:42:33.724 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-01T02:42:33.780 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-01T02:42:33.803 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 1/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-01T02:42:34.605 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-01T02:42:34.606 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-01T02:42:34.606 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-01T02:42:34.606 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T02:42:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:34.971 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:35.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-cephadm 2024-04-01T02:42:35.788 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:10 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:35.940 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 141 k 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T02:42:35.941 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:35.942 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 6.8 M 2024-04-01T02:42:35.943 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 33 M 2024-04-01T02:42:35.943 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:36.617 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-04-01T02:42:36.642 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2473.g785fc5e8.e 499 kB/s | 141 kB 00:00 2024-04-01T02:42:36.734 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 661 kB/s | 248 kB 00:00 2024-04-01T02:42:36.759 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 494 kB/s | 58 kB 00:00 2024-04-01T02:42:36.859 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-04-01T02:42:36.884 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-04-01T02:42:36.885 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:36.885 INFO:teuthology.orchestra.run.smithi187.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-04-01T02:42:36.898 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:36.908 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:36.908 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:36.998 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:36.998 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:37.301 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:37.457 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-01T02:42:37.502 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-01T02:42:38.028 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-01T02:42:38.105 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-01T02:42:38.168 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-01T02:42:38.198 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-04-01T02:42:38.958 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-04-01T02:42:38.958 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/6 2024-04-01T02:42:38.958 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-01T02:42:38.958 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-01T02:42:38.958 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-01T02:42:38.958 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T02:42:39.266 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T02:42:39.267 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T02:42:39.267 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:39.267 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:39.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2024-04-01T02:42:40.077 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:15 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 871 k 2024-04-01T02:42:40.230 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 949 k 2024-04-01T02:42:40.231 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 4.0 M 2024-04-01T02:42:40.232 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:40.369 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 571 kB/s | 79 kB 00:00 2024-04-01T02:42:40.628 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): ceph-fuse-19.0.0-2473.g785fc5e8.el9.x86_ 2.1 MB/s | 871 kB 00:00 2024-04-01T02:42:40.629 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:40.629 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.3 MB/s | 949 kB 00:00 2024-04-01T02:42:40.629 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:40.652 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:40.652 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:40.725 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:40.725 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:40.931 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:41.063 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-01T02:42:41.136 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-04-01T02:42:41.877 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-04-01T02:42:41.877 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/2 2024-04-01T02:42:42.209 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-01T02:42:42.209 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:42.209 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:42.209 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-01T02:42:42.209 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:42.209 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:42.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-volume 2024-04-01T02:42:43.055 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:18 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:43.210 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:43.210 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 264 k 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 332 k 2024-04-01T02:42:43.211 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 1.5 M 2024-04-01T02:42:43.212 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:43.324 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 616 kB/s | 69 kB 00:00 2024-04-01T02:42:43.474 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): ceph-volume-19.0.0-2473.g785fc5e8.el9.no 1.0 MB/s | 264 kB 00:00 2024-04-01T02:42:43.475 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:43.476 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-04-01T02:42:43.476 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:43.484 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:43.484 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:43.522 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:43.522 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:43.734 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:43.905 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-01T02:42:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-04-01T02:42:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-04-01T02:42:43.968 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:42:43.968 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T02:42:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 1/2 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:45.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2024-04-01T02:42:45.840 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:20 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:45.992 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 125 k 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 125 k 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 445 k 2024-04-01T02:42:45.993 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:46.315 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-19.0.0-2473.g785fc5e8.el9.x86_64 388 kB/s | 125 kB 00:00 2024-04-01T02:42:46.316 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:46.316 INFO:teuthology.orchestra.run.smithi187.stdout:Total 387 kB/s | 125 kB 00:00 2024-04-01T02:42:46.316 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:46.322 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:46.322 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:46.379 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:46.379 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:46.538 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:47.082 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:47.366 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:47.366 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:47.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2024-04-01T02:42:48.163 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:23 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:48.264 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:48.317 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:48.319 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:48.319 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:48.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2024-04-01T02:42:48.956 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:23 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:49.108 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 32 k 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:49.109 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:42:49.110 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:49.110 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 32 k 2024-04-01T02:42:49.110 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 142 k 2024-04-01T02:42:49.110 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:49.276 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-19.0.0-2473.g785fc5e8.el9.x86_6 190 kB/s | 32 kB 00:00 2024-04-01T02:42:49.277 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:49.277 INFO:teuthology.orchestra.run.smithi187.stdout:Total 188 kB/s | 32 kB 00:00 2024-04-01T02:42:49.277 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:49.281 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:49.281 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:49.310 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:49.310 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:49.400 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:49.509 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:49.913 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:50.182 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:50.182 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:50.182 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:50.183 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:50.183 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:50.183 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:50.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2024-04-01T02:42:50.925 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:25 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:51.025 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:51.077 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:51.078 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:51.078 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:51.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2024-04-01T02:42:51.720 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:26 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:51.818 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:51.869 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:51.870 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:51.870 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:51.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2024-04-01T02:42:52.505 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:27 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:52.605 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:52.657 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:52.658 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:52.658 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:52.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2024-04-01T02:42:53.288 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:28 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:53.387 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:53.439 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:53.440 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:53.440 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:53.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2024-04-01T02:42:54.082 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:29 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:54.181 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:54.232 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:54.233 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:54.233 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:54.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2024-04-01T02:42:54.875 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:29 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:54.974 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-04-01T02:42:55.027 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:55.028 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-01T02:42:55.028 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:55.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2024-04-01T02:42:55.664 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:30 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:55.814 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:55.815 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:55.815 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-04-01T02:42:55.815 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:55.815 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:55.815 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 90 k 2024-04-01T02:42:55.815 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 90 k 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 313 k 2024-04-01T02:42:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:56.121 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 294 kB/s | 90 kB 00:00 2024-04-01T02:42:56.121 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:56.122 INFO:teuthology.orchestra.run.smithi187.stdout:Total 293 kB/s | 90 kB 00:00 2024-04-01T02:42:56.122 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:56.130 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:56.130 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:56.186 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:56.186 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:42:56.329 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:42:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:42:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:57.086 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:42:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:42:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:57.086 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:42:57.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2024-04-01T02:42:57.908 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:32 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:42:58.071 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:42:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:42:58.073 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.4 M 2024-04-01T02:42:58.073 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 29 M 2024-04-01T02:42:58.073 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:42:59.883 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 1.9 MB/s | 3.4 MB 00:01 2024-04-01T02:42:59.883 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:42:59.884 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.9 MB/s | 3.4 MB 00:01 2024-04-01T02:42:59.884 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:42:59.896 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:42:59.896 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:42:59.973 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:42:59.974 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:43:00.540 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:43:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:43:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:43:00.625 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T02:43:00.625 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T02:43:00.625 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T02:43:00.626 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T02:43:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:01.905 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:43:01.906 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:01.906 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:43:01.906 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:43:01.906 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:01.906 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:43:02.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2024-04-01T02:43:02.718 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:37 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:43:02.868 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:43:02.869 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:43:02.869 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-04-01T02:43:02.869 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:43:02.869 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:43:02.869 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 176 k 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 176 k 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 673 k 2024-04-01T02:43:02.870 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:43:03.219 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 503 kB/s | 176 kB 00:00 2024-04-01T02:43:03.220 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:43:03.220 INFO:teuthology.orchestra.run.smithi187.stdout:Total 501 kB/s | 176 kB 00:00 2024-04-01T02:43:03.220 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:43:03.229 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:43:03.230 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:43:03.285 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:43:03.286 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:43:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:43:03.550 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:43:04.026 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:43:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-04-01T02:43:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:04.296 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:43:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:43:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:04.297 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:43:04.553 DEBUG:teuthology.parallel:result is None 2024-04-01T02:43:04.554 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-04-01T02:43:04.772 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T02:43:04.793 INFO:teuthology.orchestra.run.smithi187.stdout:19.0.0-2473.g785fc5e8.el9 2024-04-01T02:43:04.793 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2473.g785fc5e8.el9 2024-04-01T02:43:04.793 INFO:teuthology.task.install:The correct ceph version 19.0.0-2473.g785fc5e8 is installed. 2024-04-01T02:43:04.795 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-01T02:43:04.795 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:04.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T02:43:04.861 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-01T02:43:04.861 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:04.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T02:43:04.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T02:43:04.994 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-01T02:43:04.995 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:04.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T02:43:05.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T02:43:05.128 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-01T02:43:05.128 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:05.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T02:43:05.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T02:43:05.284 INFO:teuthology.run_tasks:Running task ceph... 2024-04-01T02:43:05.370 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-01T02:43:05.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2024-04-01T02:43:05.397 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-01T02:43:05.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-01T02:43:05.463 INFO:tasks.ceph:Creating extra log directories... 2024-04-01T02:43:05.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-01T02:43:05.539 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-01T02:43:05.539 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-01T02:43:05.539 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815', 'branch': 'ci-yingxin-crimson-load-obc-3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7633815', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'targets': {'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC7tFyiPlmPfRK7Fy3xFqDwO/tHWKJVaLJwXnBtAa8JHt21rV/oXjOPSkkgcrc2QRaO6K0jYmC3ZC+KY3lzUNI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-01_02:22:08', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-01T02:43:05.540 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-01T02:43:05.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2024-04-01T02:43:05.653 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:05.653 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-04-01T02:43:05.708 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-01T02:43:05.708 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-01 02:43:00.877620683 +0000 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-01 02:39:42.784899887 +0000 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-01 02:39:42.784899887 +0000 2024-04-01T02:43:05.763 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-01 02:39:42.784899887 +0000 2024-04-01T02:43:05.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-01T02:43:05.828 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-01T02:43:05.828 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-01T02:43:05.828 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000207986 s, 2.5 MB/s 2024-04-01T02:43:05.829 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-01T02:43:05.886 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-01 02:43:00.878620668 +0000 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-01 02:39:43.115895126 +0000 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-01 02:39:43.115895126 +0000 2024-04-01T02:43:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-01 02:39:43.115895126 +0000 2024-04-01T02:43:05.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-01T02:43:06.006 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-01T02:43:06.006 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-01T02:43:06.006 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000205626 s, 2.5 MB/s 2024-04-01T02:43:06.007 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-01T02:43:06.062 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-04-01T02:43:06.117 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-01T02:43:06.117 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T02:43:06.117 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-01T02:43:06.118 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T02:43:06.118 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T02:43:06.118 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-01 02:43:00.878620668 +0000 2024-04-01T02:43:06.118 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-01 02:39:43.450890307 +0000 2024-04-01T02:43:06.118 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-01 02:39:43.450890307 +0000 2024-04-01T02:43:06.118 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-01 02:39:43.450890307 +0000 2024-04-01T02:43:06.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-01T02:43:06.182 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-01T02:43:06.182 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-01T02:43:06.182 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000189271 s, 2.7 MB/s 2024-04-01T02:43:06.183 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-01T02:43:06.238 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-04-01T02:43:06.294 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-01T02:43:06.294 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T02:43:06.294 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-01T02:43:06.294 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T02:43:06.294 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T02:43:06.294 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-01 02:43:00.879620654 +0000 2024-04-01T02:43:06.295 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-01 02:39:43.800885272 +0000 2024-04-01T02:43:06.295 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-01 02:39:43.800885272 +0000 2024-04-01T02:43:06.295 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-01 02:39:43.800885272 +0000 2024-04-01T02:43:06.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-01T02:43:06.359 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-01T02:43:06.359 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-01T02:43:06.359 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000236537 s, 2.2 MB/s 2024-04-01T02:43:06.360 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-01T02:43:06.415 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-01T02:43:06.416 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-01T02:43:06.416 INFO:tasks.ceph:Generating config... 2024-04-01T02:43:06.417 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-01T02:43:06.417 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-01T02:43:06.417 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-01T02:43:06.417 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-01T02:43:06.417 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-01T02:43:06.418 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] log to file = False 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-01T02:43:06.419 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-01T02:43:06.419 INFO:tasks.ceph:[global] auth client required = none 2024-04-01T02:43:06.419 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-01T02:43:06.419 INFO:tasks.ceph:[global] auth service required = none 2024-04-01T02:43:06.419 INFO:tasks.ceph:[global] auth supported = none 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-01T02:43:06.420 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-01T02:43:06.421 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-01T02:43:06.422 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-01T02:43:06.423 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-01T02:43:06.424 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-01T02:43:06.424 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-01T02:43:06.424 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-01T02:43:06.425 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-01T02:43:06.425 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-01T02:43:06.425 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-01T02:43:06.425 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-01T02:43:06.425 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-01T02:43:06.425 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-01T02:43:06.425 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-01T02:43:06.425 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-01T02:43:06.425 INFO:tasks.ceph:Setting up mon.a... 2024-04-01T02:43:06.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-01T02:43:06.513 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.keyring 2024-04-01T02:43:06.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-01T02:43:06.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-01T02:43:06.595 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.187')] 2024-04-01T02:43:06.595 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.187', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-01T02:43:06.596 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:06.596 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-01T02:43:06.650 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.187 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:06.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:06.740+0000 7f7af8efaf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:06.740 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:06.740+0000 7f7af8efaf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:06.740 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:06.740+0000 7f7af8efaf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:06.740 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: generated fsid 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:setting min_mon_release = quincy 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:epoch 0 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:fsid 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:last_changed 2024-04-01T02:43:06.741603+0000 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:created 2024-04-01T02:43:06.741603+0000 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:min_mon_release 17 (quincy) 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:election_strategy: 1 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:0: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.a 2024-04-01T02:43:06.741 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-01T02:43:06.742 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-01T02:43:06.757 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f... 2024-04-01T02:43:06.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = none 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-01T02:43:06.847 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-01T02:43:06.849 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-01T02:43:06.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-01T02:43:06.851 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = 172.21.15.187 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: auth client required = none 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: auth cluster required = none 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: auth service required = none 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug asok = 0/0 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 0/0 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug bdev = 0/0 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 0/0 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 0/0 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug buffer = 0/0 2024-04-01T02:43:06.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug civetweb = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug compressor = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug context = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug crush = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug crypto = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug dpdk = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug eventtrace = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug filer = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug finisher = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug fuse = 0/0 2024-04-01T02:43:06.853 INFO:teuthology.orchestra.run.smithi187.stdout: debug heartbeatmap = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug javaclient = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug journaler = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug kstore = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug leveldb = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug lockdep = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds locker = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds log = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds log expire = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds migrator = 0/0 2024-04-01T02:43:06.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug memdb = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgrc = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug monc = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug objclass = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug objectcacher = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug objecter = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug optracker = 0/0 2024-04-01T02:43:06.855 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug perfcounter = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rados = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rbd = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rbd mirror = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rbd replay = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug refs = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw lifecycle = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw notification = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw sync = 0/0 2024-04-01T02:43:06.856 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 0/0 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: debug striper = 0/0 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: debug throttle = 0/0 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: debug timer = 0/0 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: debug tp = 0/0 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: ms client mode = crc 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: ms cluster mode = crc 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: ms mon client mode = crc 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: ms mon cluster mode = crc 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: ms mon service mode = crc 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: ms service mode = crc 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f 2024-04-01T02:43:06.857 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-01T02:43:06.858 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-01T02:43:06.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: debug monc = 20 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore compression algorithm = snappy 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore compression mode = aggressive 2024-04-01T02:43:06.860 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore rocksdb cf = False 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: crimson alien thread cpu cores = 6-7 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: crimson osd obc lru size = 10 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 0/0 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 0/0 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 0/0 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 0/0 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: err to stderr = True 2024-04-01T02:43:06.861 INFO:teuthology.orchestra.run.smithi187.stdout: log flush on exit = True 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: log to file = False 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: log to stderr = True 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: osd shutdown pgref assert = True 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.862 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 0/0 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 0/0 2024-04-01T02:43:06.863 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 0/0 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-01T02:43:06.864 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default crimson = True 2024-04-01T02:43:06.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 0-2 2024-04-01T02:43:06.866 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-04-01T02:43:06.867 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 3-5 2024-04-01T02:43:06.867 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-04-01T02:43:06.867 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 0-2 2024-04-01T02:43:06.867 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.3] 2024-04-01T02:43:06.867 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 3-5 2024-04-01T02:43:06.868 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-01T02:43:06.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-01T02:43:06.934 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-01T02:43:06.934 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:06.934 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-01T02:43:06.949 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:06.949 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-01T02:43:07.004 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:43:07.004 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:07.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-01T02:43:07.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-01T02:43:07.084 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:07.084 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:07.139 INFO:tasks.ceph:Setting up mon nodes... 2024-04-01T02:43:07.139 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-01T02:43:07.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-01T02:43:07.243 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-01T02:43:07.246 INFO:tasks.ceph:Setting up mds nodes... 2024-04-01T02:43:07.246 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-01T02:43:07.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-01T02:43:07.297 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-01T02:43:07.313 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-01T02:43:07.314 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-01T02:43:07.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-01T02:43:07.340 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-01T02:43:07.340 INFO:tasks.ceph:role: osd.0 2024-04-01T02:43:07.340 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:43:07.340 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-01T02:43:07.414 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T02:43:07.415 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T02:43:07.934 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-01T02:43:07.935 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-01T02:43:07.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-01T02:43:08.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-01T02:43:08.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-01T02:43:08.172 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-01T02:43:08.172 INFO:tasks.ceph:role: osd.1 2024-04-01T02:43:08.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:43:08.172 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-01T02:43:08.246 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T02:43:08.247 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T02:43:08.247 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T02:43:08.247 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T02:43:08.699 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-01T02:43:08.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-01T02:43:08.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-01T02:43:08.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-01T02:43:08.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-01T02:43:08.897 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-01T02:43:08.897 INFO:tasks.ceph:role: osd.2 2024-04-01T02:43:08.897 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:43:08.897 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-01T02:43:08.972 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T02:43:08.972 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T02:43:08.972 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T02:43:08.972 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T02:43:08.972 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T02:43:08.973 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-01T02:43:08.973 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T02:43:08.973 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T02:43:08.973 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T02:43:08.973 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T02:43:09.470 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-01T02:43:09.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-01T02:43:09.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-01T02:43:09.581 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-01T02:43:09.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:09.912 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T02:43:09.913 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-04-01 02:43:09,915 seastar - Reactor backend: linux-aio 2024-04-01T02:43:09.924 INFO:teuthology.orchestra.run.smithi187.stderr:==29191==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T02:43:10.030 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x5d8c926 0x5d945ff 0x5d946b9 0x5ddaaa1 0x5ddaca9 0x5ddae89 0x5d59970 0x5d59b4a 0x5d5a0d6 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x3785266 0x3785dbf 0x3625d7a 0x3feaf 0x3ff5f 0x3412444 2024-04-01T02:43:10.030 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-04-01T02:43:10.056 INFO:teuthology.orchestra.run.smithi187.stderr:DEBUG 2024-04-01 02:43:10,058 [shard 0:main] seastar - reactor::drain 2024-04-01T02:43:10.627 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:10.627 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-04-01T02:43:10.627 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-04-01T02:43:10.627 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-04-01T02:43:10.627 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:10.627 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:10.681 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T02:43:10.685 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-04-01 02:43:10,686 seastar - Reactor backend: linux-aio 2024-04-01T02:43:10.693 INFO:teuthology.orchestra.run.smithi187.stderr:==29189==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:15.340 INFO:teuthology.orchestra.run.smithi187.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f 2024-04-01T02:43:15.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-01T02:43:15.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T02:43:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-04-01 02:43:15,680 seastar - Reactor backend: linux-aio 2024-04-01T02:43:15.684 INFO:teuthology.orchestra.run.smithi187.stderr:==29373==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T02:43:15.817 INFO:teuthology.orchestra.run.smithi187.stderr:DEBUG 2024-04-01 02:43:15,818 [shard 0:main] seastar - reactor::drain 2024-04-01T02:43:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-04-01T02:43:16.347 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-04-01T02:43:16.347 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-04-01T02:43:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:16.347 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:16.409 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T02:43:16.412 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-04-01 02:43:16,414 seastar - Reactor backend: linux-aio 2024-04-01T02:43:16.429 INFO:teuthology.orchestra.run.smithi187.stderr:==29371==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T02:43:21.068 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:21.068 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-04-01T02:43:21.068 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-04-01T02:43:21.068 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-04-01T02:43:21.068 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:21.069 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:21.069 INFO:teuthology.orchestra.run.smithi187.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f 2024-04-01T02:43:21.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-01T02:43:21.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:21.404 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T02:43:21.405 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-04-01 02:43:21,407 seastar - Reactor backend: linux-aio 2024-04-01T02:43:21.410 INFO:teuthology.orchestra.run.smithi187.stderr:==29555==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T02:43:21.542 INFO:teuthology.orchestra.run.smithi187.stderr:DEBUG 2024-04-01 02:43:21,543 [shard 0:main] seastar - reactor::drain 2024-04-01T02:43:22.077 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:22.077 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-04-01T02:43:22.077 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-04-01T02:43:22.077 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-04-01T02:43:22.077 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:22.077 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:22.137 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T02:43:22.140 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-04-01 02:43:22,142 seastar - Reactor backend: linux-aio 2024-04-01T02:43:22.149 INFO:teuthology.orchestra.run.smithi187.stderr:==29553==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T02:43:26.812 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:26.812 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-04-01T02:43:26.812 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-04-01T02:43:26.812 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-04-01T02:43:26.812 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-04-01T02:43:26.812 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:26.813 INFO:teuthology.orchestra.run.smithi187.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f 2024-04-01T02:43:26.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-01T02:43:26.862 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-01T02:43:26.862 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:26.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-01T02:43:26.929 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:26.929 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-01T02:43:26.996 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:26.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-01T02:43:27.062 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:27.063 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-01T02:43:27.129 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:27.130 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-01T02:43:27.184 INFO:tasks.ceph:Adding keys to all mons... 2024-04-01T02:43:27.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.x] 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQA7HwpmKF2aDhAAW2EaOPqieMJlUKSI7HrJfQ== 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQA+HwpmIEdWMhAA8zhi9EsAd7kPx30cSkYIig== 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQBEHwpmU7x3IhAA9bvlpEEfSKwdq9Gie59CIQ== 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQBKHwpmZGLqERAARwhFY+QFlx4/Ms71MKoE2g== 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2024-04-01T02:43:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQA7Hwpmjo3OERAAhk2wYUXuUKGXwXtNWqrf6g== 2024-04-01T02:43:27.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-01T02:43:27.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T02:43:27.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T02:43:27.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T02:43:27.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-01T02:43:27.604 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-01T02:43:27.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-01T02:43:27.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-01T02:43:27.746 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:27.747+0000 7f0367692a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:27.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:27.747+0000 7f0367692a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:27.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-01T02:43:27.786 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-01T02:43:27.840 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-01T02:43:27.841 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-01T02:43:27.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-01T02:43:27.883 INFO:tasks.ceph.mon.a:Started 2024-04-01T02:43:27.884 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-01T02:43:27.884 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-01T02:43:27.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-01T02:43:27.886 INFO:tasks.ceph.mgr.x:Started 2024-04-01T02:43:27.886 INFO:tasks.ceph:Setting crush tunables to default 2024-04-01T02:43:27.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd crush tunables default 2024-04-01T02:43:27.968 INFO:tasks.ceph.mon.a.smithi187.stderr:2024-04-01T02:43:27.969+0000 7f63f862ba80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:27.968 INFO:tasks.ceph.mon.a.smithi187.stderr:2024-04-01T02:43:27.969+0000 7f63f862ba80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:27.968 INFO:tasks.ceph.mon.a.smithi187.stderr:2024-04-01T02:43:27.969+0000 7f63f862ba80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:27.970 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:27.971+0000 7f085e1c7000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:27.994+0000 7faa0dc13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:28.001 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:28.002+0000 7faa0dc13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:28.070 INFO:teuthology.orchestra.run.smithi187.stderr:adjusted tunables profile to default 2024-04-01T02:43:28.081 INFO:tasks.ceph:check_enable_crimson: True 2024-04-01T02:43:28.082 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-01T02:43:28.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-01T02:43:28.173 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:28.173+0000 7f085e1c7000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:28.173 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:28.174+0000 7f085e1c7000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:28.192 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:28.188+0000 7fe7d6413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:28.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:28.194+0000 7fe7d6413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:28.272 INFO:tasks.ceph.mon.a.smithi187.stderr:2024-04-01T02:43:28.273+0000 7f63eeff4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-01T02:43:28.272 INFO:tasks.ceph.mon.a.smithi187.stderr:Please be aware that this feature is experimental, untested, 2024-04-01T02:43:28.272 INFO:tasks.ceph.mon.a.smithi187.stderr:unsupported, and may result in data corruption, data loss, 2024-04-01T02:43:28.272 INFO:tasks.ceph.mon.a.smithi187.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-01T02:43:28.272 INFO:tasks.ceph.mon.a.smithi187.stderr:feature with important data. 2024-04-01T02:43:28.272 INFO:tasks.ceph.mon.a.smithi187.stderr: 2024-04-01T02:43:28.651 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:28.651+0000 7f085e1c7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T02:43:28.705 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:28.705+0000 7f085e1c7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T02:43:29.082 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-01T02:43:29.083 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:29.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-01T02:43:29.110 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:29.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-01T02:43:29.176 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:29.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-01T02:43:29.243 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.244+0000 7f085e1c7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T02:43:29.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 976c8e8b-72c6-430c-b601-aced1a122144 0 2024-04-01T02:43:29.299 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.300+0000 7f085e1c7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T02:43:29.346 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.347+0000 7f085e1c7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T02:43:29.394 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:29.389+0000 7f41527f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:29.396 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:29.397+0000 7f41527f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:29.405 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.406+0000 7f085e1c7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T02:43:29.449 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.450+0000 7f085e1c7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T02:43:29.474 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:29.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 0ba95431-cfa5-46fa-a177-33e40abd0dbb 1 2024-04-01T02:43:29.571 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.571+0000 7f085e1c7000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T02:43:29.636 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:29.632+0000 7f0aee954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:29.639 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:29.639+0000 7f0aee954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:29.646 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.647+0000 7f085e1c7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T02:43:29.693 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.694+0000 7f085e1c7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T02:43:29.717 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-01T02:43:29.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 21ca2222-b516-40e7-9d49-3ef01e599239 2 2024-04-01T02:43:29.749 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.750+0000 7f085e1c7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T02:43:29.799 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:29.800+0000 7f085e1c7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T02:43:29.840 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:29.835+0000 7f0ae5ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:29.841 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:29.842+0000 7f0ae5ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:29.919 INFO:teuthology.orchestra.run.smithi187.stdout:2 2024-04-01T02:43:29.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-01T02:43:30.003 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-01T02:43:30.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-01T02:43:30.045 INFO:tasks.ceph.osd.0:Started 2024-04-01T02:43:30.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-01T02:43:30.076 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-01T02:43:30.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-01T02:43:30.118 INFO:tasks.ceph.osd.1:Started 2024-04-01T02:43:30.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-01T02:43:30.150 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-01T02:43:30.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-01T02:43:30.192 INFO:tasks.ceph.osd.2:Started 2024-04-01T02:43:30.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T02:43:30.212 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.213+0000 7f085e1c7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T02:43:30.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:30.319+0000 7fe98ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:30.325 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:30.326+0000 7fe98ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:30.369 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.370+0000 7f085e1c7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T02:43:30.396 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:30.396 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"fsid":"50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f","created":"2024-04-01T02:43:27.987657+0000","modified":"2024-04-01T02:43:29.920621+0000","last_up_change":"0.000000","last_in_change":"2024-04-01T02:43:29.920621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"976c8e8b-72c6-430c-b601-aced1a122144","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0ba95431-cfa5-46fa-a177-33e40abd0dbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"21ca2222-b516-40e7-9d49-3ef01e599239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T02:43:30.409 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-01T02:43:30.409 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-01T02:43:30.523 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.524+0000 7f085e1c7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T02:43:30.573 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.573+0000 7f085e1c7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T02:43:30.693 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.694+0000 7f085e1c7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T02:43:30.711 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-01T02:43:30.800 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.800+0000 7f085e1c7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T02:43:30.813 INFO:teuthology.misc.health.smithi187.stderr:2024-04-01T02:43:30.810+0000 7f1f8d660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:30.815 INFO:teuthology.misc.health.smithi187.stderr:2024-04-01T02:43:30.816+0000 7f1f8d660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:30.884 INFO:teuthology.misc.health.smithi187.stdout: 2024-04-01T02:43:30.884 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":6,"fsid":"50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f","created":"2024-04-01T02:43:27.987657+0000","modified":"2024-04-01T02:43:29.920621+0000","last_up_change":"0.000000","last_in_change":"2024-04-01T02:43:29.920621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"976c8e8b-72c6-430c-b601-aced1a122144","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0ba95431-cfa5-46fa-a177-33e40abd0dbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"21ca2222-b516-40e7-9d49-3ef01e599239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T02:43:30.895 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-01T02:43:30.973 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:30.973+0000 7f085e1c7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T02:43:31.020 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:31.021+0000 7f085e1c7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T02:43:31.192 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:31.192+0000 7f085e1c7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T02:43:32.078 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:32.079+0000 7f085e1c7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T02:43:32.363 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:32.364+0000 7f085e1c7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T02:43:32.417 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:32.418+0000 7f085e1c7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T02:43:32.610 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:32.611+0000 7f085e1c7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T02:43:33.621 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-01T02:43:33.622+0000 7f07fd426640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-01T02:43:37.198 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-01T02:43:37.311 INFO:teuthology.misc.health.smithi187.stderr:2024-04-01T02:43:37.308+0000 7f8140f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:37.313 INFO:teuthology.misc.health.smithi187.stderr:2024-04-01T02:43:37.314+0000 7f8140f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:37.499 INFO:teuthology.misc.health.smithi187.stdout: 2024-04-01T02:43:37.499 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":9,"fsid":"50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f","created":"2024-04-01T02:43:27.987657+0000","modified":"2024-04-01T02:43:36.627056+0000","last_up_change":"2024-04-01T02:43:36.627056+0000","last_in_change":"2024-04-01T02:43:29.920621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"976c8e8b-72c6-430c-b601-aced1a122144","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3886846340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":3886846340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6805","nonce":3886846340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3886846340}]},"public_addr":"172.21.15.187:6808/3886846340","cluster_addr":"172.21.15.187:6809/3886846340","heartbeat_back_addr":"172.21.15.187:6805/3886846340","heartbeat_front_addr":"172.21.15.187:6804/3886846340","state":["exists","up"]},{"osd":1,"uuid":"0ba95431-cfa5-46fa-a177-33e40abd0dbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2738990358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":2738990358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":2738990358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":2738990358}]},"public_addr":"172.21.15.187:6812/2738990358","cluster_addr":"172.21.15.187:6811/2738990358","heartbeat_back_addr":"172.21.15.187:6803/2738990358","heartbeat_front_addr":"172.21.15.187:6802/2738990358","state":["exists","up"]},{"osd":2,"uuid":"21ca2222-b516-40e7-9d49-3ef01e599239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2648754575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":2648754575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":2648754575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":2648754575}]},"public_addr":"172.21.15.187:6816/2648754575","cluster_addr":"172.21.15.187:6817/2648754575","heartbeat_back_addr":"172.21.15.187:6807/2648754575","heartbeat_front_addr":"172.21.15.187:6806/2648754575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T02:43:37.510 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-01T02:43:37.511 INFO:tasks.ceph:Creating RBD pool 2024-04-01T02:43:37.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-01T02:43:37.622 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:37.616+0000 7f535f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:37.624 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:37.624+0000 7f535f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:38.633 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'rbd' created 2024-04-01T02:43:38.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-01T02:43:38.818 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:38.807+0000 7f8a7ce24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:38.820 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:38.821+0000 7f8a7ce24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:39.640 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-01T02:43:39.658 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-01T02:43:39.659 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-01T02:43:39.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-01T02:43:39.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-01T02:43:39.659 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-01T02:43:39.806 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:39.804+0000 7f19d842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:39.808 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:39.809+0000 7f19d842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.042 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:40.053 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1161087271},{"type":"v1","addr":"172.21.15.187:6801","nonce":1161087271}]},"active_addr":"172.21.15.187:6801/1161087271","active_change":"2024-04-01T02:43:32.613856+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":104441423}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3619095745}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":4105508549}]}]} 2024-04-01T02:43:40.057 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-01T02:43:40.057 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-01T02:43:40.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T02:43:40.167 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.164+0000 7f6f561c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.168 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.169+0000 7f6f561c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.352 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:40.352 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":12,"fsid":"50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f","created":"2024-04-01T02:43:27.987657+0000","modified":"2024-04-01T02:43:39.638721+0000","last_up_change":"2024-04-01T02:43:36.627056+0000","last_in_change":"2024-04-01T02:43:29.920621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T02:43:36.636522+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-01T02:43:37.833553+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"976c8e8b-72c6-430c-b601-aced1a122144","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3886846340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":3886846340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6805","nonce":3886846340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3886846340}]},"public_addr":"172.21.15.187:6808/3886846340","cluster_addr":"172.21.15.187:6809/3886846340","heartbeat_back_addr":"172.21.15.187:6805/3886846340","heartbeat_front_addr":"172.21.15.187:6804/3886846340","state":["exists","up"]},{"osd":1,"uuid":"0ba95431-cfa5-46fa-a177-33e40abd0dbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2738990358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":2738990358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":2738990358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":2738990358}]},"public_addr":"172.21.15.187:6812/2738990358","cluster_addr":"172.21.15.187:6811/2738990358","heartbeat_back_addr":"172.21.15.187:6803/2738990358","heartbeat_front_addr":"172.21.15.187:6802/2738990358","state":["exists","up"]},{"osd":2,"uuid":"21ca2222-b516-40e7-9d49-3ef01e599239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2648754575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":2648754575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":2648754575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":2648754575}]},"public_addr":"172.21.15.187:6816/2648754575","cluster_addr":"172.21.15.187:6817/2648754575","heartbeat_back_addr":"172.21.15.187:6807/2648754575","heartbeat_front_addr":"172.21.15.187:6806/2648754575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T02:43:40.365 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-01T02:43:40.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T02:43:40.478 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.475+0000 7fc3dbcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.480 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.481+0000 7fc3dbcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.668 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:40.668 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"fsid":"50a9a515-aff5-4bc2-92fe-e7d24ffb6e3f","created":"2024-04-01T02:43:27.987657+0000","modified":"2024-04-01T02:43:40.642242+0000","last_up_change":"2024-04-01T02:43:36.627056+0000","last_in_change":"2024-04-01T02:43:29.920621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T02:43:36.636522+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-01T02:43:37.833553+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"976c8e8b-72c6-430c-b601-aced1a122144","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3886846340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":3886846340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6805","nonce":3886846340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3886846340}]},"public_addr":"172.21.15.187:6808/3886846340","cluster_addr":"172.21.15.187:6809/3886846340","heartbeat_back_addr":"172.21.15.187:6805/3886846340","heartbeat_front_addr":"172.21.15.187:6804/3886846340","state":["exists","up"]},{"osd":1,"uuid":"0ba95431-cfa5-46fa-a177-33e40abd0dbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2738990358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":2738990358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":2738990358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":2738990358}]},"public_addr":"172.21.15.187:6812/2738990358","cluster_addr":"172.21.15.187:6811/2738990358","heartbeat_back_addr":"172.21.15.187:6803/2738990358","heartbeat_front_addr":"172.21.15.187:6802/2738990358","state":["exists","up"]},{"osd":2,"uuid":"21ca2222-b516-40e7-9d49-3ef01e599239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2648754575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":2648754575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":2648754575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":2648754575}]},"public_addr":"172.21.15.187:6816/2648754575","cluster_addr":"172.21.15.187:6817/2648754575","heartbeat_back_addr":"172.21.15.187:6807/2648754575","heartbeat_front_addr":"172.21.15.187:6806/2648754575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T02:43:40.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-01T02:43:40.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-01T02:43:40.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-01T02:43:40.800 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.796+0000 7f7a48ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.800 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.796+0000 7f55029e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.800 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.796+0000 7f3d8ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.802 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.803+0000 7f55029e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.802 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.803+0000 7f3d8ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.802 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.803+0000 7f7a48ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.827 INFO:teuthology.orchestra.run.smithi187.stdout:34359738373 2024-04-01T02:43:40.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-01T02:43:40.834 INFO:teuthology.orchestra.run.smithi187.stdout:38654705668 2024-04-01T02:43:40.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-01T02:43:40.835 INFO:teuthology.orchestra.run.smithi187.stdout:34359738373 2024-04-01T02:43:40.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-01T02:43:40.943 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.939+0000 7f9abbe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.945 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.946+0000 7f9abbe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.993 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.989+0000 7f3d2746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.994 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.991+0000 7fa5af66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.995 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.996+0000 7f3d2746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:40.996 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:40.997+0000 7fa5af66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:41.144 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:41.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-01T02:43:41.209 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:41.212 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:41.223 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-01T02:43:41.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-01T02:43:42.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-01T02:43:42.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-01T02:43:42.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-01T02:43:42.286 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:42.282+0000 7fa7cc51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:42.289 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:42.290+0000 7fa7cc51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:42.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:42.339+0000 7fa8c8c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:42.347 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:42.347+0000 7fa8c8c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:42.387 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:42.383+0000 7fb6d0154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:42.389 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:42.390+0000 7fb6d0154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:42.484 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:42.498 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-01T02:43:42.554 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:42.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-01T02:43:42.599 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:42.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-01T02:43:43.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-01T02:43:43.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-01T02:43:43.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-01T02:43:43.617 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:43.614+0000 7f4758c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:43.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:43.621+0000 7f4758c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:43.690 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:43.686+0000 7ff948d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:43.693 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:43.694+0000 7ff948d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:43.731 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:43.727+0000 7f483422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:43.733 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:43.734+0000 7f483422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:43.822 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:43.835 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-01T02:43:43.890 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:43.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-01T02:43:43.925 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:43.938 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-01T02:43:44.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-01T02:43:44.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-01T02:43:44.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-01T02:43:44.962 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:44.957+0000 7f1fb615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:44.964+0000 7f1fb615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:45.027 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:45.020+0000 7fa0fadf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:45.029 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:45.029+0000 7fa0fadf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:45.063 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:45.059+0000 7f2fc7ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:45.065 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:45.065+0000 7f2fc7ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:45.185 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:45.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-01T02:43:45.228 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:45.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-01T02:43:45.280 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-01T02:43:45.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-01T02:43:46.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-01T02:43:46.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-01T02:43:46.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-01T02:43:46.319 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.314+0000 7f7e6e4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.322 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.321+0000 7f7e6e4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.359+0000 7f912fe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.366+0000 7f912fe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.408+0000 7faa5135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.415 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.415+0000 7faa5135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.524 INFO:teuthology.orchestra.run.smithi187.stdout:34359738377 2024-04-01T02:43:46.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738377 for osd.1 2024-04-01T02:43:46.539 DEBUG:teuthology.parallel:result is None 2024-04-01T02:43:46.563 INFO:teuthology.orchestra.run.smithi187.stdout:38654705668 2024-04-01T02:43:46.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-04-01T02:43:46.576 DEBUG:teuthology.parallel:result is None 2024-04-01T02:43:46.609 INFO:teuthology.orchestra.run.smithi187.stdout:34359738377 2024-04-01T02:43:46.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738377 for osd.0 2024-04-01T02:43:46.622 DEBUG:teuthology.parallel:result is None 2024-04-01T02:43:46.622 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-01T02:43:46.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T02:43:46.774 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.771+0000 7f2b6135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.776 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:46.776+0000 7f2b6135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:46.959 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:46.959 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2024-04-01T02:43:46.971 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-01T02:43:46.618761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80924,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034596,"statfs":{"total":289910292480,"available":289827426304,"internally_reserved":0,"allocated":438272,"data_stored":1254537,"data_compressed":59916,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.960183"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.665342+0000","last_change":"2024-04-01T02:43:39.761075+0000","last_active":"2024-04-01T02:43:40.665342+0000","last_peered":"2024-04-01T02:43:40.665342+0000","last_clean":"2024-04-01T02:43:40.665342+0000","last_became_active":"2024-04-01T02:43:39.758989+0000","last_became_peered":"2024-04-01T02:43:39.758989+0000","last_unstale":"2024-04-01T02:43:40.665342+0000","last_undegraded":"2024-04-01T02:43:40.665342+0000","last_fullsized":"2024-04-01T02:43:40.665342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.665541+0000","last_change":"2024-04-01T02:43:39.740653+0000","last_active":"2024-04-01T02:43:40.665541+0000","last_peered":"2024-04-01T02:43:40.665541+0000","last_clean":"2024-04-01T02:43:40.665541+0000","last_became_active":"2024-04-01T02:43:39.738040+0000","last_became_peered":"2024-04-01T02:43:39.738040+0000","last_unstale":"2024-04-01T02:43:40.665541+0000","last_undegraded":"2024-04-01T02:43:40.665541+0000","last_fullsized":"2024-04-01T02:43:40.665541+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.662176+0000","last_change":"2024-04-01T02:43:39.751315+0000","last_active":"2024-04-01T02:43:40.662176+0000","last_peered":"2024-04-01T02:43:40.662176+0000","last_clean":"2024-04-01T02:43:40.662176+0000","last_became_active":"2024-04-01T02:43:39.748358+0000","last_became_peered":"2024-04-01T02:43:39.748358+0000","last_unstale":"2024-04-01T02:43:40.662176+0000","last_undegraded":"2024-04-01T02:43:40.662176+0000","last_fullsized":"2024-04-01T02:43:40.662176+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.666076+0000","last_change":"2024-04-01T02:43:39.743042+0000","last_active":"2024-04-01T02:43:40.666076+0000","last_peered":"2024-04-01T02:43:40.666076+0000","last_clean":"2024-04-01T02:43:40.666076+0000","last_became_active":"2024-04-01T02:43:39.740656+0000","last_became_peered":"2024-04-01T02:43:39.740656+0000","last_unstale":"2024-04-01T02:43:40.666076+0000","last_undegraded":"2024-04-01T02:43:40.666076+0000","last_fullsized":"2024-04-01T02:43:40.666076+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T02:43:39.719141+0000","last_change":"2024-04-01T02:43:39.719389+0000","last_active":"2024-04-01T02:43:39.719141+0000","last_peered":"2024-04-01T02:43:39.719141+0000","last_clean":"2024-04-01T02:43:38.633403+0000","last_became_active":"2024-04-01T02:43:39.717158+0000","last_became_peered":"2024-04-01T02:43:39.717158+0000","last_unstale":"2024-04-01T02:43:39.719141+0000","last_undegraded":"2024-04-01T02:43:39.719141+0000","last_fullsized":"2024-04-01T02:43:39.719141+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:39.691958+0000","last_change":"2024-04-01T02:43:39.692286+0000","last_active":"2024-04-01T02:43:39.691958+0000","last_peered":"2024-04-01T02:43:39.691958+0000","last_clean":"2024-04-01T02:43:38.633403+0000","last_became_active":"2024-04-01T02:43:39.688974+0000","last_became_peered":"2024-04-01T02:43:39.688974+0000","last_unstale":"2024-04-01T02:43:39.691958+0000","last_undegraded":"2024-04-01T02:43:39.691958+0000","last_fullsized":"2024-04-01T02:43:39.691958+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:39.707969+0000","last_change":"2024-04-01T02:43:39.708325+0000","last_active":"2024-04-01T02:43:39.707969+0000","last_peered":"2024-04-01T02:43:39.707969+0000","last_clean":"2024-04-01T02:43:38.633403+0000","last_became_active":"2024-04-01T02:43:39.702610+0000","last_became_peered":"2024-04-01T02:43:39.702610+0000","last_unstale":"2024-04-01T02:43:39.707969+0000","last_undegraded":"2024-04-01T02:43:39.707969+0000","last_fullsized":"2024-04-01T02:43:39.707969+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.665905+0000","last_change":"2024-04-01T02:43:39.695525+0000","last_active":"2024-04-01T02:43:40.665905+0000","last_peered":"2024-04-01T02:43:40.665905+0000","last_clean":"2024-04-01T02:43:40.665905+0000","last_became_active":"2024-04-01T02:43:39.693100+0000","last_became_peered":"2024-04-01T02:43:39.693100+0000","last_unstale":"2024-04-01T02:43:40.665905+0000","last_undegraded":"2024-04-01T02:43:40.665905+0000","last_fullsized":"2024-04-01T02:43:40.665905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.663404+0000","last_change":"2024-04-01T02:43:38.702464+0000","last_active":"2024-04-01T02:43:40.663404+0000","last_peered":"2024-04-01T02:43:40.663404+0000","last_clean":"2024-04-01T02:43:40.663404+0000","last_became_active":"2024-04-01T02:43:38.697994+0000","last_became_peered":"2024-04-01T02:43:38.697994+0000","last_unstale":"2024-04-01T02:43:40.663404+0000","last_undegraded":"2024-04-01T02:43:40.663404+0000","last_fullsized":"2024-04-01T02:43:40.663404+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:37.632741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:37.632741+0000","last_clean_scrub_stamp":"2024-04-01T02:43:37.632741+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":23636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616897,"data_compressed":29958,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614004,"data_compressed":29958,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T02:43:46.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T02:43:47.083 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:47.080+0000 7f7952b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:47.085 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:47.085+0000 7f7952b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:47.268 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:47.269 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2024-04-01T02:43:47.280 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-01T02:43:46.618761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80924,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034596,"statfs":{"total":289910292480,"available":289827426304,"internally_reserved":0,"allocated":438272,"data_stored":1254537,"data_compressed":59916,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.960183"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.665342+0000","last_change":"2024-04-01T02:43:39.761075+0000","last_active":"2024-04-01T02:43:40.665342+0000","last_peered":"2024-04-01T02:43:40.665342+0000","last_clean":"2024-04-01T02:43:40.665342+0000","last_became_active":"2024-04-01T02:43:39.758989+0000","last_became_peered":"2024-04-01T02:43:39.758989+0000","last_unstale":"2024-04-01T02:43:40.665342+0000","last_undegraded":"2024-04-01T02:43:40.665342+0000","last_fullsized":"2024-04-01T02:43:40.665342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.665541+0000","last_change":"2024-04-01T02:43:39.740653+0000","last_active":"2024-04-01T02:43:40.665541+0000","last_peered":"2024-04-01T02:43:40.665541+0000","last_clean":"2024-04-01T02:43:40.665541+0000","last_became_active":"2024-04-01T02:43:39.738040+0000","last_became_peered":"2024-04-01T02:43:39.738040+0000","last_unstale":"2024-04-01T02:43:40.665541+0000","last_undegraded":"2024-04-01T02:43:40.665541+0000","last_fullsized":"2024-04-01T02:43:40.665541+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.662176+0000","last_change":"2024-04-01T02:43:39.751315+0000","last_active":"2024-04-01T02:43:40.662176+0000","last_peered":"2024-04-01T02:43:40.662176+0000","last_clean":"2024-04-01T02:43:40.662176+0000","last_became_active":"2024-04-01T02:43:39.748358+0000","last_became_peered":"2024-04-01T02:43:39.748358+0000","last_unstale":"2024-04-01T02:43:40.662176+0000","last_undegraded":"2024-04-01T02:43:40.662176+0000","last_fullsized":"2024-04-01T02:43:40.662176+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.666076+0000","last_change":"2024-04-01T02:43:39.743042+0000","last_active":"2024-04-01T02:43:40.666076+0000","last_peered":"2024-04-01T02:43:40.666076+0000","last_clean":"2024-04-01T02:43:40.666076+0000","last_became_active":"2024-04-01T02:43:39.740656+0000","last_became_peered":"2024-04-01T02:43:39.740656+0000","last_unstale":"2024-04-01T02:43:40.666076+0000","last_undegraded":"2024-04-01T02:43:40.666076+0000","last_fullsized":"2024-04-01T02:43:40.666076+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T02:43:39.719141+0000","last_change":"2024-04-01T02:43:39.719389+0000","last_active":"2024-04-01T02:43:39.719141+0000","last_peered":"2024-04-01T02:43:39.719141+0000","last_clean":"2024-04-01T02:43:38.633403+0000","last_became_active":"2024-04-01T02:43:39.717158+0000","last_became_peered":"2024-04-01T02:43:39.717158+0000","last_unstale":"2024-04-01T02:43:39.719141+0000","last_undegraded":"2024-04-01T02:43:39.719141+0000","last_fullsized":"2024-04-01T02:43:39.719141+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:39.691958+0000","last_change":"2024-04-01T02:43:39.692286+0000","last_active":"2024-04-01T02:43:39.691958+0000","last_peered":"2024-04-01T02:43:39.691958+0000","last_clean":"2024-04-01T02:43:38.633403+0000","last_became_active":"2024-04-01T02:43:39.688974+0000","last_became_peered":"2024-04-01T02:43:39.688974+0000","last_unstale":"2024-04-01T02:43:39.691958+0000","last_undegraded":"2024-04-01T02:43:39.691958+0000","last_fullsized":"2024-04-01T02:43:39.691958+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:39.707969+0000","last_change":"2024-04-01T02:43:39.708325+0000","last_active":"2024-04-01T02:43:39.707969+0000","last_peered":"2024-04-01T02:43:39.707969+0000","last_clean":"2024-04-01T02:43:38.633403+0000","last_became_active":"2024-04-01T02:43:39.702610+0000","last_became_peered":"2024-04-01T02:43:39.702610+0000","last_unstale":"2024-04-01T02:43:39.707969+0000","last_undegraded":"2024-04-01T02:43:39.707969+0000","last_fullsized":"2024-04-01T02:43:39.707969+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.665905+0000","last_change":"2024-04-01T02:43:39.695525+0000","last_active":"2024-04-01T02:43:40.665905+0000","last_peered":"2024-04-01T02:43:40.665905+0000","last_clean":"2024-04-01T02:43:40.665905+0000","last_became_active":"2024-04-01T02:43:39.693100+0000","last_became_peered":"2024-04-01T02:43:39.693100+0000","last_unstale":"2024-04-01T02:43:40.665905+0000","last_undegraded":"2024-04-01T02:43:40.665905+0000","last_fullsized":"2024-04-01T02:43:40.665905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:38.633403+0000","last_clean_scrub_stamp":"2024-04-01T02:43:38.633403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T02:43:40.663404+0000","last_change":"2024-04-01T02:43:38.702464+0000","last_active":"2024-04-01T02:43:40.663404+0000","last_peered":"2024-04-01T02:43:40.663404+0000","last_clean":"2024-04-01T02:43:40.663404+0000","last_became_active":"2024-04-01T02:43:38.697994+0000","last_became_peered":"2024-04-01T02:43:38.697994+0000","last_unstale":"2024-04-01T02:43:40.663404+0000","last_undegraded":"2024-04-01T02:43:40.663404+0000","last_fullsized":"2024-04-01T02:43:40.663404+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T02:43:37.632741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T02:43:37.632741+0000","last_clean_scrub_stamp":"2024-04-01T02:43:37.632741+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":23636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616897,"data_compressed":29958,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614004,"data_compressed":29958,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T02:43:47.281 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-01T02:43:47.281 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-01T02:43:47.281 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-01T02:43:47.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-01T02:43:47.390 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:47.388+0000 7fd0a27f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:47.392 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:43:47.392+0000 7fd0a27f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:43:47.614 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:47.614 INFO:teuthology.orchestra.run.smithi187.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-01T02:43:47.627 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-01T02:43:47.627 INFO:teuthology.run_tasks:Running task cbt... 2024-04-01T02:43:47.641 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi187.front.sepia.ceph.com', 'osds': ['ubuntu@smithi187.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi187.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi187.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi187.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-01T02:43:47.641 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-01T02:43:47.663 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:43:47.663 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-01T02:43:47.714 DEBUG:teuthology.orchestra.run.smithi187:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-01T02:43:47.769 INFO:teuthology.orchestra.run.smithi187.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-01T02:43:48.117 INFO:tasks.cbt:Installing collectl 2024-04-01T02:43:48.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-01T02:43:48.223 INFO:teuthology.orchestra.run.smithi187.stderr:--2024-04-01 02:43:48-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-01T02:43:48.375 INFO:teuthology.orchestra.run.smithi187.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-04-01T02:43:48.384 INFO:teuthology.orchestra.run.smithi187.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-04-01T02:43:48.607 INFO:teuthology.orchestra.run.smithi187.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-01T02:43:48.607 INFO:teuthology.orchestra.run.smithi187.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmCh9kKiYrnwoAs7jpyZZ-O5Du5VIDzNNgEx6SWhQyWsA8b6YwHFssBb80kc13aVp3VjQxDgy2muZnlg5l9dkxBFTDaQ%3D%3D&use_mirror=gigenet&r= [following] 2024-04-01T02:43:48.607 INFO:teuthology.orchestra.run.smithi187.stderr:--2024-04-01 02:43:48-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmCh9kKiYrnwoAs7jpyZZ-O5Du5VIDzNNgEx6SWhQyWsA8b6YwHFssBb80kc13aVp3VjQxDgy2muZnlg5l9dkxBFTDaQ%3D%3D&use_mirror=gigenet&r= 2024-04-01T02:43:48.607 INFO:teuthology.orchestra.run.smithi187.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-01T02:43:48.672 INFO:teuthology.orchestra.run.smithi187.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-01T02:43:48.923 INFO:teuthology.orchestra.run.smithi187.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-01T02:43:48.923 INFO:teuthology.orchestra.run.smithi187.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-01T02:43:48.923 INFO:teuthology.orchestra.run.smithi187.stderr:--2024-04-01 02:43:48-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-01T02:43:48.935 INFO:teuthology.orchestra.run.smithi187.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2024-04-01T02:43:48.966 INFO:teuthology.orchestra.run.smithi187.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2024-04-01T02:43:49.142 INFO:teuthology.orchestra.run.smithi187.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-01T02:43:49.142 INFO:teuthology.orchestra.run.smithi187.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-01T02:43:49.142 INFO:teuthology.orchestra.run.smithi187.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-01T02:43:49.142 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:49.190 INFO:teuthology.orchestra.run.smithi187.stderr: 0K .......... .......... .......... .......... .......... 8% 1.00M 1s 2024-04-01T02:43:49.222 INFO:teuthology.orchestra.run.smithi187.stderr: 50K .......... .......... .......... .......... .......... 16% 1.61M 0s 2024-04-01T02:43:49.222 INFO:teuthology.orchestra.run.smithi187.stderr: 100K .......... .......... .......... .......... .......... 24% 45.6M 0s 2024-04-01T02:43:49.233 INFO:teuthology.orchestra.run.smithi187.stderr: 150K .......... .......... .......... .......... .......... 32% 4.00M 0s 2024-04-01T02:43:49.255 INFO:teuthology.orchestra.run.smithi187.stderr: 200K .......... .......... .......... .......... .......... 41% 2.75M 0s 2024-04-01T02:43:49.256 INFO:teuthology.orchestra.run.smithi187.stderr: 250K .......... .......... .......... .......... .......... 49% 99.0M 0s 2024-04-01T02:43:49.256 INFO:teuthology.orchestra.run.smithi187.stderr: 300K .......... .......... .......... .......... .......... 57% 53.0M 0s 2024-04-01T02:43:49.264 INFO:teuthology.orchestra.run.smithi187.stderr: 350K .......... .......... .......... .......... .......... 65% 4.24M 0s 2024-04-01T02:43:49.281 INFO:teuthology.orchestra.run.smithi187.stderr: 400K .......... .......... .......... .......... .......... 74% 2.80M 0s 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr: 450K .......... .......... .......... .......... .......... 82% 14.9M 0s 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr: 500K .......... .......... .......... .......... .......... 90% 219M 0s 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr: 550K .......... .......... .......... .......... .......... 98% 396M 0s 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr: 600K ...... 100% 11.4T=0.1s 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01 02:43:49 (4.11 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-01T02:43:49.286 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:43:49.333 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/ 2024-04-01T02:43:49.333 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/collectl.conf 2024-04-01T02:43:49.333 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/ 2024-04-01T02:43:49.333 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-01T02:43:49.333 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Socket.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/colmux.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Import.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/style.css 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Process.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-01T02:43:49.334 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Export.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Startup.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Performance.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Messages.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Logging.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-01T02:43:49.335 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Misc.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Examples.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Data.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-01T02:43:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Playback.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Hello.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Network.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Memory.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/Disks.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/service/ 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/service/collectl.service 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/initd/ 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-01T02:43:49.337 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/initd/collectl 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/misc.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/gexpr.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/statsd.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/envrules.std 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/hello.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/vmsum.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/proctree.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/README-WINDOWS 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/formatit.ph 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/colmux 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/INSTALL 2024-04-01T02:43:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/graphite.ph 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/vmstat.ph 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/README 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/man1/ 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/man1/collectl.1 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/man1/colmux.1 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/lexpr.ph 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/client.pl 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/collectl 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/COPYING 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/vnet.ph 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/GPL 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/UNINSTALL 2024-04-01T02:43:49.339 INFO:teuthology.orchestra.run.smithi187.stdout:collectl-4.3.1/ARTISTIC 2024-04-01T02:43:49.381 INFO:teuthology.orchestra.run.smithi187.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-01T02:43:49.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-01T02:43:49.961 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:24 ago on Mon 01 Apr 2024 02:40:25 AM UTC. 2024-04-01T02:43:50.063 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-01T02:43:50.066 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-01T02:43:50.124 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-01T02:43:50.124 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: librbd-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 23 k 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: perf x86_64 5.14.0-434.el9 CentOS-AppStream 2.2 M 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: libradospp-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 30 k 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-01T02:43:50.125 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-01T02:43:50.126 INFO:teuthology.orchestra.run.smithi187.stdout:Install 5 Packages 2024-04-01T02:43:50.126 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:50.126 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.3 M 2024-04-01T02:43:50.126 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 11 M 2024-04-01T02:43:50.126 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-01T02:43:50.370 INFO:teuthology.orchestra.run.smithi187.stdout:(1/5): librbd-devel-19.0.0-2473.g785fc5e8.el9.x 121 kB/s | 23 kB 00:00 2024-04-01T02:43:50.412 INFO:teuthology.orchestra.run.smithi187.stdout:(2/5): perf-5.14.0-434.el9.x86_64.rpm 9.3 MB/s | 2.2 MB 00:00 2024-04-01T02:43:50.454 INFO:teuthology.orchestra.run.smithi187.stdout:(3/5): libradospp-devel-19.0.0-2473.g785fc5e8.e 111 kB/s | 30 kB 00:00 2024-04-01T02:43:50.646 INFO:teuthology.orchestra.run.smithi187.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 56 kB/s | 13 kB 00:00 2024-04-01T02:43:50.746 INFO:teuthology.orchestra.run.smithi187.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 316 kB/s | 118 kB 00:00 2024-04-01T02:43:50.749 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-01T02:43:50.749 INFO:teuthology.orchestra.run.smithi187.stdout:Total 3.8 MB/s | 2.3 MB 00:00 2024-04-01T02:43:50.749 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-01T02:43:50.767 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-01T02:43:50.768 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-01T02:43:50.881 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-01T02:43:50.881 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-01T02:43:51.190 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-01T02:43:51.249 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-01T02:43:51.279 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-01T02:43:51.310 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/5 2024-04-01T02:43:51.619 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/5 2024-04-01T02:43:51.708 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : perf-5.14.0-434.el9.x86_64 5/5 2024-04-01T02:43:52.639 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: perf-5.14.0-434.el9.x86_64 5/5 2024-04-01T02:43:52.639 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/5 2024-04-01T02:43:52.639 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/5 2024-04-01T02:43:52.639 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : perf-5.14.0-434.el9.x86_64 3/5 2024-04-01T02:43:52.639 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: pdsh-2.34-7.el9.x86_64 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: perf-5.14.0-434.el9.x86_64 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-01T02:43:52.974 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-01T02:43:53.287 INFO:tasks.cbt:benchmark: radosbench 2024-04-01T02:43:53.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-01T02:43:53.315 DEBUG:teuthology.orchestra.run.smithi187:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-01T02:43:53.713 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - Settings.cluster: 2024-04-01T02:43:53.713 INFO:teuthology.orchestra.run.smithi187.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-01T02:43:53.713 INFO:teuthology.orchestra.run.smithi187.stderr: 'clients': ['ubuntu@smithi187.front.sepia.ceph.com'], 2024-04-01T02:43:53.713 INFO:teuthology.orchestra.run.smithi187.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-01T02:43:53.713 INFO:teuthology.orchestra.run.smithi187.stderr: 'head': 'ubuntu@smithi187.front.sepia.ceph.com', 2024-04-01T02:43:53.713 INFO:teuthology.orchestra.run.smithi187.stderr: 'is_teuthology': True, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'iterations': 1, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'mons': ['ubuntu@smithi187.front.sepia.ceph.com'], 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'osds': ['ubuntu@smithi187.front.sepia.ceph.com'], 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'osds_per_node': 3, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'pid_dir': '/var/run/ceph', 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'pgp_size': 256, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'replication': 'replicated'}}, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'rebuild_every_test': False, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'rgws': {'ubuntu@smithi187.front.sepia.ceph.com': None}, 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-01T02:43:53.714 INFO:teuthology.orchestra.run.smithi187.stderr: 'use_existing': True, 2024-04-01T02:43:53.715 INFO:teuthology.orchestra.run.smithi187.stderr: 'user': 'ubuntu'} 2024-04-01T02:43:53.733 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-04-01T02:43:53.734 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:43:53.735 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:43:53.735 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-04-01T02:43:53.738 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:43:53.738 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-01T02:43:53.740 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:43:53.740 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:43:53.740 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-04-01T02:43:53.742 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:43:53.742 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-01T02:43:53.743 INFO:teuthology.orchestra.run.smithi187.stderr:02:43:53 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-01T02:44:53.775 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.775 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-01T02:44:53.815 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.815 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-01T02:44:53.853 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.853 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-04-01T02:44:53.855 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.855 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-04-01T02:44:53.857 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.857 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.857 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-04-01T02:44:53.874 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.874 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/$(basename ${i}).smithi187.front.sepia.ceph.com ; done 2024-04-01T02:44:53.878 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.878 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.878 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-04-01T02:44:53.898 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - WARNING - cbt - sudo killall -9 rados 2024-04-01T02:44:53.898 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - WARNING - cbt - error 1 seen, continuing anyway... 2024-04-01T02:44:53.918 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c. 2024-04-01T02:44:53.918 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-04-01T02:44:53.918 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.918 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.918 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-04-01T02:44:53.934 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.934 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.934 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-04-01T02:44:53.937 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.937 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-01T02:44:53.939 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.939 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.939 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-04-01T02:44:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-01T02:44:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-01T02:44:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-04-01T02:44:54.253 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:54 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:54.253 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:54 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:54.253 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-04-01T02:44:54.952 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:54 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:44:54.952 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:54 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:44:54.952 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-04-01T02:44:55.955 INFO:teuthology.orchestra.run.smithi187.stderr:02:44:55 - INFO - cbt - Checking Health after pool creation. 2024-04-01T02:45:00.961 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:00 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:00.961 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:00 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:00.961 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-01T02:45:01.451 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:01 - INFO - cbt - Final Pool Health Check. 2024-04-01T02:45:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-01T02:45:06.802 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.803 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-01T02:45:06.820 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.820 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-01T02:45:06.850 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.850 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-04-01T02:45:06.852 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.852 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-04-01T02:45:06.854 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.854 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:06.854 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-01T02:45:07.418 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:07 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:07.419 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-01T02:45:08.007 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:08.007 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:08.007 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-01T02:45:08.368 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:08.368 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:08.368 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-04-01T02:45:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-04-01T02:45:09.234 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.234 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.234 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-04-01T02:45:09.236 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.236 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-01T02:45:09.238 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.238 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.238 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-04-01T02:45:09.241 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.241 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-01T02:45:09.241 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-01T02:45:09.241 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - INFO - cbt - Running radosbench write test. 2024-04-01T02:45:09.241 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.242 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.242 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-04-01T02:45:09.243 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.243 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:45:09.243 INFO:teuthology.orchestra.run.smithi187.stderr:02:45:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-04-01T02:46:10.399 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:10.399 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-01T02:46:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-01T02:46:10.764 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:10.764 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-04-01T02:46:10.766 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:10.766 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-04-01T02:46:10.768 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:10.768 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:10.768 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-04-01T02:46:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:11.106 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:11.106 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-04-01T02:46:11.122 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:11.123 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/$(basename ${i}).smithi187.front.sepia.ceph.com ; done 2024-04-01T02:46:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - INFO - cbt - Convert results to json format. 2024-04-01T02:46:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-01T02:46:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:12 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-01T02:46:12.670 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:12 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:12.670 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:12 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:12.670 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-01T02:46:13.039 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:13.039 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:13.039 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand 2024-04-01T02:46:13.110 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:13.110 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:13.110 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/ceph_settings.out 2024-04-01T02:46:13.997 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:13.997 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:13.998 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl 2024-04-01T02:46:14.000 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.000 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-01T02:46:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - INFO - cbt - Running radosbench rand test. 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.0 2024-04-01T02:46:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:46:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:02:46:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.1 2024-04-01T02:47:15.172 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.173 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-01T02:47:15.269 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.269 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-01T02:47:15.302 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.302 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf.data 2024-04-01T02:47:15.303 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.304 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.* 2024-04-01T02:47:15.305 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.305 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.305 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/historic_ops.out 2024-04-01T02:47:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.963 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.964 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* 2024-04-01T02:47:15.981 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - get_fqdn_local()=smithi187.front.sepia.ceph.com 2024-04-01T02:47:15.981 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/$(basename ${i}).smithi187.front.sepia.ceph.com ; done 2024-04-01T02:47:16.055 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:16 - INFO - cbt - Convert results to json format. 2024-04-01T02:47:16.055 INFO:teuthology.orchestra.run.smithi187.stderr:02:47:16 - DEBUG - cbt - Nodes : ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:16.235 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-01T02:47:16.244 DEBUG:teuthology.orchestra.run.smithi187:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-01T02:47:17.595 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-01T02:47:17.595 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-01T02:47:17.596 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-01T02:47:17.674 INFO:teuthology.orchestra.run.smithi187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi187.front.sepia.ceph.com 2024-04-01T02:47:17.674 INFO:teuthology.orchestra.run.smithi187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi187.front.sepia.ceph.com 2024-04-01T02:47:17.674 INFO:teuthology.orchestra.run.smithi187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi187.front.sepia.ceph.com 2024-04-01T02:47:17.674 INFO:teuthology.orchestra.run.smithi187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi187.front.sepia.ceph.com 2024-04-01T02:47:17.675 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:47:17.675 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi187.front.sepia.ceph.com of=/dev/stdout 2024-04-01T02:47:17.702 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-01T02:47:17.702 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2024-04-01T02:47:17.760 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-01T02:47:17.760 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-01T02:47:17.760 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:47:17.760 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi187.front.sepia.ceph.com of=/dev/stdout 2024-04-01T02:47:17.827 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-01T02:47:17.827 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2024-04-01T02:47:17.884 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-01T02:47:17.885 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-01T02:47:17.885 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:47:17.885 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi187.front.sepia.ceph.com of=/dev/stdout 2024-04-01T02:47:17.944 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-01T02:47:17.944 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2024-04-01T02:47:18.001 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-01T02:47:18.001 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-01T02:47:18.001 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-01T02:47:18.001 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi187.front.sepia.ceph.com of=/dev/stdout 2024-04-01T02:47:18.058 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-01T02:47:18.059 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2024-04-01T02:47:18.115 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-01T02:47:18.115 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-01T02:47:18.125 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-01T02:47:18.132 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-01T02:47:18.138 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-01T02:47:18.147 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-01T02:47:18.147 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-01T02:47:18.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-01T02:47:18.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:47:18.275+0000 7fc0d6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:47:18.284 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-01T02:47:18.284+0000 7fc0d6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T02:47:18.492 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-01T02:47:18.492 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-01T02:47:18.493 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-01T02:47:18.493 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T02:47:20.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-01T02:47:24.594 INFO:tasks.ceph.osd.0:Stopped 2024-04-01T02:47:24.595 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-01T02:47:24.595 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T02:47:25.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-01T02:47:25.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-01T02:47:30.697 INFO:tasks.ceph.osd.1:Stopped 2024-04-01T02:47:30.697 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-01T02:47:30.697 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T02:47:30.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-01T02:47:36.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-01T02:47:36.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-01T02:47:36.799 INFO:tasks.ceph.osd.2:Stopped 2024-04-01T02:47:36.800 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-01T02:47:36.800 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-01T02:47:36.800 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T02:47:36.833 INFO:tasks.ceph.mgr.x:Stopped 2024-04-01T02:47:36.833 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-01T02:47:36.833 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-01T02:47:36.833 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T02:47:36.890 INFO:tasks.ceph.mon.a:Stopped 2024-04-01T02:47:36.890 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-01T02:47:36.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-01T02:47:37.002 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:37.002 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-01T02:47:37.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:37.220 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-01T02:47:37.308 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:37.308 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-01T02:47:37.374 INFO:tasks.ceph:Archiving mon data... 2024-04-01T02:47:37.470 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815/data/mon.a.tgz 2024-04-01T02:47:37.471 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-01T02:47:37.497 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.L75kOruDaL 2024-04-01T02:47:37.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.L75kOruDaL 2024-04-01T02:47:37.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.L75kOruDaL 2024-04-01T02:47:37.724 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.L75kOruDaL is 192KB 2024-04-01T02:47:37.778 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.L75kOruDaL 2024-04-01T02:47:37.792 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-01T02:47:37.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-01T02:47:37.914 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-01T02:47:37.914 INFO:tasks.ceph:Archiving crash dumps... 2024-04-01T02:47:37.916 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815/remote/smithi187/crash 2024-04-01T02:47:37.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-01T02:47:37.980 INFO:tasks.ceph:Compressing logs... 2024-04-01T02:47:37.980 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-01T02:47:38.078 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31399.log 2024-04-01T02:47:38.078 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-01T02:47:38.078 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31612.log 2024-04-01T02:47:38.078 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30164.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31272.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31672.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31360.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31798.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31108.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31005.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31798.log: /var/log/ceph/ceph-client.admin.31108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31400.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31150.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30314.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31923.log 2024-04-01T02:47:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31150.log: /var/log/ceph/ceph-client.admin.30314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30542.log 2024-04-01T02:47:38.080 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31990.log 2024-04-01T02:47:38.080 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30949.log 2024-04-01T02:47:38.089 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31632.log 2024-04-01T02:47:38.089 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30512.log 2024-04-01T02:47:38.089 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31632.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-01T02:47:38.089 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32855.log 2024-04-01T02:47:38.089 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32074.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33160.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32074.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.33160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33460.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.33160.log.gz 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32718.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.33460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31192.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33460.log.gz 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31274.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31273.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31864.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30354.log 2024-04-01T02:47:38.090 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31864.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30354.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28717.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32809.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.tmp-client.admin.28717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31546.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30076.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32809.log: /var/log/ceph/ceph-client.admin.31546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31758.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.30076.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32919.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31884.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31506.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32032.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31506.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32032.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31738.log 2024-04-01T02:47:38.091 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31486.log 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32832.log 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31047.log 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33159.log 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.31047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32920.log 2024-04-01T02:47:38.092 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.33159.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.33159.log.gz 2024-04-01T02:47:38.096 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.32920.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31399.log.gz 2024-04-01T02:47:38.096 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31612.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31632.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.31672.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31864.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31798.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30164.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30314.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30542.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30512.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32718.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32855.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30354.log.gz 2024-04-01T02:47:38.097 INFO:teuthology.orchestra.run.smithi187.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28717.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32809.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30076.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.32919.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.32832.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31005.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30949.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31047.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31192.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31272.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31360.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31273.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31486.log.gz 2024-04-01T02:47:38.098 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31150.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31108.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31400.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31506.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31274.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31546.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31738.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.31758.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31884.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31923.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31990.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32032.log.gz 2024-04-01T02:47:38.099 INFO:teuthology.orchestra.run.smithi187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32074.log.gz 2024-04-01T02:47:38.100 INFO:teuthology.orchestra.run.smithi187.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-01T02:47:38.105 INFO:teuthology.orchestra.run.smithi187.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-01T02:47:38.109 INFO:teuthology.orchestra.run.smithi187.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.32920.log.gz 2024-04-01T02:47:38.292 INFO:teuthology.orchestra.run.smithi187.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-01T02:47:41.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-01T02:47:47.846 INFO:teuthology.orchestra.run.smithi187.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-01T02:47:48.274 INFO:teuthology.orchestra.run.smithi187.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-01T02:47:48.724 INFO:teuthology.orchestra.run.smithi187.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-01T02:47:48.725 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-01T02:47:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m10.690s 2024-04-01T02:47:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m8.355s 2024-04-01T02:47:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m0.452s 2024-04-01T02:47:48.726 INFO:tasks.ceph:Archiving logs... 2024-04-01T02:47:48.726 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815/remote/smithi187/log 2024-04-01T02:47:48.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T02:47:49.950 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-01T02:47:49.959 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-01T02:47:49.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T02:47:50.023 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-01T02:47:50.032 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-01T02:47:50.033 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T02:47:50.050 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-01T02:47:50.102 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T02:47:50.102 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-01T02:47:50.102 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 -437us[ -437us] +/- 85ms 2024-04-01T02:47:50.102 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -679us[ -680us] +/- 60ms 2024-04-01T02:47:50.102 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +1087us[+1086us] +/- 62ms 2024-04-01T02:47:50.102 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:47:50.123 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-01T02:47:50.132 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-01T02:47:50.133 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-01T02:47:50.153 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T02:47:50.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T02:47:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:47:50.301 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-01T02:47:50.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-01T02:47:50.333 INFO:teuthology.task.internal:Duration was 728.969782 seconds 2024-04-01T02:47:50.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-01T02:47:50.361 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-01T02:47:50.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T02:47:50.492 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T02:47:50.619 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-01T02:47:50.619 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-04-01T02:47:50.619 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T02:47:50.638 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-01T02:47:50.638 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T02:47:50.706 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-01T02:47:50.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T02:47:50.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-01T02:47:50.865 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-01T02:47:50.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T02:47:50.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-01T02:47:50.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:47:50.993 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-04-01T02:47:51.011 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:47:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:47:51.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-01T02:47:51.036 INFO:teuthology.task.internal:Transferring archived files... 2024-04-01T02:47:51.036 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-01_02:22:08-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7633815/remote/smithi187 2024-04-01T02:47:51.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T02:47:51.148 INFO:teuthology.task.internal:Removing archive directory... 2024-04-01T02:47:51.148 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T02:47:51.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-01T02:47:51.208 INFO:teuthology.task.internal:Not uploading archives. 2024-04-01T02:47:51.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-01T02:47:51.246 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-01T02:47:51.246 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T02:47:51.261 INFO:teuthology.orchestra.run.smithi187.stdout: 266040 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 02:47 /home/ubuntu/cephtest 2024-04-01T02:47:51.262 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-01T02:47:51.279 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-01T02:47:51.348 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=ci-yingxin-crimson-load-obc-3&var-job_id_selected=7633815 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} duration: 728.9697818756104 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-04-01T02:47:51.348 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T02:47:51.428 INFO:teuthology.run:pass