2024-03-29T02:09:11.839 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T02:09:11.840 DEBUG:teuthology.run:Teuthology command: teuthology --name yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi --archive /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066 --owner scheduled_yingxin@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} --verbose -- /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066/orig.config.yaml 2024-03-29T02:09:11.898 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T02:09:11.986 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066 branch: ci-yingxin-crimson-load-obc-3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7629066' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-3 env: CRIMSON_COMPAT: '1' sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErUD6NxtA2tbihaaJOJjdayyeu7io1ALB2tFTs1euHgmOHRFVVazHU+TXcng8iY850x09JMjQGUuX0tSuiFLUU= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_01:54:41 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T02:09:11.986 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa; will attempt to use it 2024-03-29T02:09:11.987 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks 2024-03-29T02:09:11.987 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T02:09:11.989 INFO:teuthology.task.internal:Checking packages... 2024-03-29T02:09:12.012 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '785fc5e8d093374f751c7da3e51e9b56b3718d17' 2024-03-29T02:09:12.013 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T02:09:12.013 INFO:teuthology.packaging:ref: None 2024-03-29T02:09:12.013 INFO:teuthology.packaging:tag: None 2024-03-29T02:09:12.013 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-3 2024-03-29T02:09:12.013 INFO:teuthology.packaging:sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:09:12.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-3 2024-03-29T02:09:12.154 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2473.g785fc5e8 2024-03-29T02:09:12.156 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T02:09:12.162 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T02:09:12.162 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T02:09:12.174 INFO:teuthology.task.internal:Saving configuration 2024-03-29T02:09:12.191 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T02:09:12.197 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T02:09:12.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 02:04:13.731424', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErUD6NxtA2tbihaaJOJjdayyeu7io1ALB2tFTs1euHgmOHRFVVazHU+TXcng8iY850x09JMjQGUuX0tSuiFLUU='} 2024-03-29T02:09:12.223 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T02:09:12.266 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-29T02:09:12.266 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T02:09:12.306 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faaf4c6af70>, signals=[15]) 2024-03-29T02:09:12.306 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T02:09:12.313 INFO:teuthology.task.internal:Opening connections... 2024-03-29T02:09:12.313 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:09:12.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:09:12.405 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T02:09:12.412 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-03-29T02:09:12.477 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-03-29T02:09:12.478 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-03-29T02:09:12.534 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-03-29T02:09:12.534 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-03-29T02:09:12.534 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-03-29T02:09:12.534 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-03-29T02:09:12.534 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-03-29T02:09:12.535 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T02:09:12.536 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T02:09:12.536 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T02:09:12.536 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-03-29T02:09:12.564 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T02:09:12.572 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T02:09:12.577 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T02:09:12.578 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-03-29T02:09:12.593 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T02:09:12.600 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T02:09:12.600 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-03-29T02:09:12.656 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T02:09:12.693 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T02:09:12.709 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T02:09:12.709 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T02:09:12.709 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T02:09:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:09:12.725 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-03-29T02:09:12.781 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-361.el9.x86_64 2024-03-29T02:09:12.782 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-361.el9.x86_64 2024-03-29T02:09:12.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-03-29T02:09:15.152 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.7 MB 00:00 2024-03-29T02:09:16.853 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T02:09:18.193 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-03-29T02:09:21.694 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-03-29T02:09:26.645 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-29T02:09:28.170 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-29T02:09:30.905 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-03-29T02:09:35.888 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T02:09:38.162 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T02:09:38.162 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T02:09:38.246 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:09:38.259 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:09:38.260 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:09:38.260 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-03-29T02:09:38.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:09:38.260 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 76 M 2024-03-29T02:09:38.260 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 123 M 2024-03-29T02:09:38.260 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:09:38.431 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T02:09:39.023 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T02:09:39.526 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T02:09:40.065 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T02:09:40.068 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:09:40.068 INFO:teuthology.orchestra.run.smithi137.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-29T02:09:40.069 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:09:40.443 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:09:40.443 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:09:41.326 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:09:41.326 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:09:43.639 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:09:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T02:09:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:09:47.097 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:09:47.233 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:09:51.911 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:09:51.957 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:09:56.710 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:10:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:10:25.872 INFO:teuthology.orchestra.run.smithi137.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T02:10:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:10:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:10:26.317 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:10:26.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T02:10:26.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:10:26.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:10:26.735 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:10:26.736 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:10:27.070 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T02:10:27.163 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T02:10:27.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T02:10:27.164 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T02:10:27.164 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T02:10:27.164 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T02:10:27.164 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T02:10:27.164 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T02:10:27.164 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T02:10:27.245 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T02:10:27.245 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T02:10:27.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-03-29T02:10:27.279 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-03-29T02:10:27.279 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-03-29T02:10:27.279 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-03-29T02:10:27.279 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-03-29T02:10:27.280 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:10:27.281 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T02:10:27.282 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T02:10:27.282 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-434.el9.x86_64 2024-03-29T02:10:27.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T02:10:27.937 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-03-29T02:10:29.353 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T02:10:29.384 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-03-29T02:10:29.387 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-03-29T02:10:29.442 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.pV3R6TwAsr 2024-03-29T02:10:29.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pV3R6TwAsr 2024-03-29T02:10:29.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.pV3R6TwAsr 2024-03-29T02:10:29.630 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.pV3R6TwAsr is 6KB 2024-03-29T02:10:29.678 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.pV3R6TwAsr 2024-03-29T02:10:29.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T02:10:29.758 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T02:10:29.758 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T02:10:29.758 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T02:10:29.758 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T02:10:29.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T02:10:29.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-03-29T02:10:59.930 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T02:10:59.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T02:10:59.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:11:18.431 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-03-29T02:11:27.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T02:11:27.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:11:30.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-03-29T02:11:42.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T02:11:42.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:12:14.338 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-03-29T02:12:14.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T02:12:14.988 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T02:12:14.988 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-03-29T02:12:15.018 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-434.el9.x86_64 2024-03-29T02:12:15.019 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T02:12:15.019 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T02:12:15.019 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T02:12:16.020 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T02:12:16.020 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T02:12:16.106 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-03-29T02:12:16.136 DEBUG:teuthology.parallel:result is None 2024-03-29T02:12:16.137 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T02:12:16.145 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T02:12:16.146 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T02:12:16.161 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T02:12:16.167 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T02:12:16.177 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T02:12:16.177 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T02:12:16.293 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T02:12:16.300 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T02:12:16.300 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T02:12:16.404 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T02:12:16.417 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T02:12:16.419 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T02:12:16.426 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T02:12:16.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T02:12:16.452 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T02:12:16.461 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T02:12:16.461 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T02:12:16.510 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T02:12:16.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T02:12:16.645 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T02:12:16.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T02:12:16.776 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:12:16.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T02:12:16.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-03-29T02:12:16.936 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T02:12:17.497 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T02:12:17.506 INFO:teuthology.task.internal:Starting timer... 2024-03-29T02:12:17.506 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T02:12:17.522 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T02:12:17.531 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-29T02:12:17.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-03-29T02:12:17.592 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-03-29T02:12:17.595 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T02:12:17.595 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-03-29T02:12:17.615 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-03-29T02:12:17.615 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-03-29T02:12:17.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T02:12:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:12:17.686 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T02:12:17.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-03-29T02:12:17.767 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T02:12:17.777 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T02:12:17.796 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T02:12:17.797 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T02:16:29.212 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-03-29T02:16:29.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T02:16:29.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:16:29.305 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-03-29T02:16:29.390 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T02:16:29.390 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T02:16:29.401 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T02:16:29.402 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T02:16:29.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T02:16:29.461 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T02:16:29.478 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T02:16:29.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-03-29T02:16:29.520 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-03-29T02:16:29.537 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T02:16:29.554 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T02:16:29.610 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-03-29T02:16:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T02:16:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-03-29T02:16:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:16:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:16:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:16:29.614 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:16:29.614 INFO:teuthology.run_tasks:Running task install... 2024-03-29T02:16:29.622 DEBUG:teuthology.task.install:project ceph 2024-03-29T02:16:29.622 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}} 2024-03-29T02:16:29.622 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'} 2024-03-29T02:16:29.622 INFO:teuthology.task.install:Using flavor: crimson 2024-03-29T02:16:29.628 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T02:16:29.628 INFO:teuthology.task.install:extra packages: [] 2024-03-29T02:16:29.628 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'tag': None, 'wait_for_package': False} 2024-03-29T02:16:29.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:16:29.894 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/ 2024-03-29T02:16:29.894 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2473.g785fc5e8 2024-03-29T02:16:30.064 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T02:16:30.065 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:16:30.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T02:16:30.094 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T02:16:30.094 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/785fc5e8d093374f751c7da3e51e9b56b3718d17/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T02:16:30.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T02:16:30.242 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T02:16:30.308 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-03-29T02:16:30.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-03-29T02:16:30.558 INFO:teuthology.orchestra.run.smithi137.stdout:74 files removed 2024-03-29T02:16:30.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-03-29T02:16:31.356 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 185 kB/s | 87 kB 00:00 2024-03-29T02:16:31.757 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-03-29T02:16:32.082 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 7.1 kB/s | 2.0 kB 00:00 2024-03-29T02:16:32.518 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T02:16:33.960 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-29T02:16:36.492 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-29T02:16:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T02:16:42.894 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T02:16:44.270 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-29T02:16:47.121 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 21 MB 00:02 2024-03-29T02:16:52.129 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-29T02:16:54.770 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:16:54.786 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:16:54.786 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:16:54.786 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 9.6 M 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.7 M 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.9 M 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 19 M 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 25 k 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T02:16:54.787 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 796 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 547 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse 2024-03-29T02:16:54.788 INFO:teuthology.orchestra.run.smithi137.stdout: x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 45 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 134 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 159 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 315 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 293 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 99 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:16:54.789 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:16:54.790 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:16:54.790 INFO:teuthology.orchestra.run.smithi137.stdout:Install 30 Packages 2024-03-29T02:16:54.790 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-03-29T02:16:54.790 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:16:54.790 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 58 M 2024-03-29T02:16:54.790 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:16:56.505 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-common-19.0.0-2473.g785fc5e8.el9.x 12 MB/s | 19 MB 00:01 2024-03-29T02:16:56.564 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-selinux-19.0.0-2473.g785fc5e8.el9. 433 kB/s | 25 kB 00:00 2024-03-29T02:16:56.680 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): ceph-base-19.0.0-2473.g785fc5e8.el9.x86 3.5 MB/s | 5.9 MB 00:01 2024-03-29T02:16:56.722 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): libcephfs2-19.0.0-2473.g785fc5e8.el9.x8 4.9 MB/s | 796 kB 00:00 2024-03-29T02:16:56.831 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): libradosstriper1-19.0.0-2473.g785fc5e8. 3.6 MB/s | 547 kB 00:00 2024-03-29T02:16:56.948 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-ceph-argparse-19.0.0-2473.g785f 386 kB/s | 45 kB 00:00 2024-03-29T02:16:57.073 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): librgw2-19.0.0-2473.g785fc5e8.el9.x86_6 14 MB/s | 5.0 MB 00:00 2024-03-29T02:16:57.098 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-ceph-common-19.0.0-2473.g785fc5 891 kB/s | 134 kB 00:00 2024-03-29T02:16:57.215 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): ceph-radosgw-19.0.0-2473.g785fc5e8.el9. 4.3 MB/s | 9.6 MB 00:02 2024-03-29T02:16:57.240 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-cephfs-19.0.0-2473.g785fc5e8.e 953 kB/s | 159 kB 00:00 2024-03-29T02:16:57.266 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-rados-19.0.0-2473.g785fc5e8.el 1.8 MB/s | 315 kB 00:00 2024-03-29T02:16:57.291 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-rgw-19.0.0-2473.g785fc5e8.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T02:16:57.341 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-rbd-19.0.0-2473.g785fc5e8.el9. 2.3 MB/s | 293 kB 00:00 2024-03-29T02:16:57.436 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 625 kB/s | 106 kB 00:00 2024-03-29T02:16:57.461 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 471 kB/s | 44 kB 00:00 2024-03-29T02:16:57.486 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 933 kB/s | 159 kB 00:00 2024-03-29T02:16:57.561 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 661 kB 00:00 2024-03-29T02:16:57.587 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-03-29T02:16:57.612 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 329 kB/s | 41 kB 00:00 2024-03-29T02:16:57.637 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T02:16:57.664 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-03-29T02:16:57.688 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 616 kB/s | 46 kB 00:00 2024-03-29T02:16:57.812 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-29T02:16:57.921 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-29T02:16:57.980 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 902 kB/s | 309 kB 00:00 2024-03-29T02:16:58.022 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-29T02:16:58.164 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-29T02:16:58.189 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T02:16:58.273 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-03-29T02:16:58.323 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-29T02:16:58.756 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): librbd1-19.0.0-2473.g785fc5e8.el9.x86_ 7.6 MB/s | 3.7 MB 00:00 2024-03-29T02:16:58.815 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): librados2-19.0.0-2473.g785fc5e8.el9.x8 5.4 MB/s | 3.4 MB 00:00 2024-03-29T02:16:58.819 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:16:58.819 INFO:teuthology.orchestra.run.smithi137.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T02:16:58.820 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:16:58.855 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:16:58.855 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:16:59.334 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:16:59.335 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:17:00.871 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:17:01.334 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T02:17:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:17:01.449 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:17:01.540 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-29T02:17:01.568 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:17:01.630 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:17:01.674 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-29T02:17:01.706 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-29T02:17:01.731 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-29T02:17:01.761 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-29T02:17:01.777 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-29T02:17:01.945 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-29T02:17:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-29T02:17:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-29T02:17:02.374 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:17:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:17:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 12/34 2024-03-29T02:17:02.540 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-29T02:17:02.581 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 14/34 2024-03-29T02:17:02.685 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-29T02:17:02.728 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-29T02:17:02.815 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-29T02:17:02.881 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-29T02:17:02.995 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-29T02:17:03.044 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-29T02:17:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-29T02:17:03.125 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-29T02:17:03.154 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-29T02:17:03.189 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-29T02:17:03.234 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T02:17:03.767 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T02:17:03.792 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-29T02:17:03.851 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-29T02:17:03.922 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 27/34 2024-03-29T02:17:03.961 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 28/34 2024-03-29T02:17:06.329 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:17:06.366 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:17:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:17:07.077 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-29T02:17:07.153 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-29T02:17:07.153 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T02:17:07.153 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T02:17:07.153 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:07.187 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-03-29T02:17:17.167 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-03-29T02:17:17.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:18.181 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-29T02:17:18.205 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-29T02:17:18.205 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:17:18.205 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T02:17:18.205 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T02:17:18.205 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T02:17:18.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:18.207 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T02:17:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T02:17:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/34 2024-03-29T02:17:19.525 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 6/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 7/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 8/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 9/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 12/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T02:17:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T02:17:19.527 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:17:19.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T02:17:19.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/34 2024-03-29T02:17:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:17:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:20.634 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-03-29T02:17:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T02:17:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:20.637 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:17:20.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-03-29T02:17:21.511 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:17:21.669 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:17:21.670 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:17:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 87 M 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 174 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T02:17:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout:Install 11 Packages 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 90 M 2024-03-29T02:17:21.672 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 819 M 2024-03-29T02:17:21.673 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:17:22.155 INFO:teuthology.orchestra.run.smithi137.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T02:17:22.313 INFO:teuthology.orchestra.run.smithi137.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T02:17:22.372 INFO:teuthology.orchestra.run.smithi137.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T02:17:22.397 INFO:teuthology.orchestra.run.smithi137.stdout:(4/11): libcephsqlite-19.0.0-2473.g785fc5e8.el9 435 kB/s | 174 kB 00:00 2024-03-29T02:17:22.414 INFO:teuthology.orchestra.run.smithi137.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T02:17:22.431 INFO:teuthology.orchestra.run.smithi137.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.7 MB/s | 90 kB 00:00 2024-03-29T02:17:22.506 INFO:teuthology.orchestra.run.smithi137.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-29T02:17:23.007 INFO:teuthology.orchestra.run.smithi137.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 374 kB/s | 187 kB 00:00 2024-03-29T02:17:23.115 INFO:teuthology.orchestra.run.smithi137.stdout:(9/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 685 kB/s | 469 kB 00:00 2024-03-29T02:17:23.224 INFO:teuthology.orchestra.run.smithi137.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 562 kB/s | 122 kB 00:00 2024-03-29T02:17:27.165 INFO:teuthology.orchestra.run.smithi137.stdout:(11/11): ceph-test-19.0.0-2473.g785fc5e8.el9.x8 17 MB/s | 87 MB 00:05 2024-03-29T02:17:27.168 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:17:27.168 INFO:teuthology.orchestra.run.smithi137.stdout:Total 16 MB/s | 90 MB 00:05 2024-03-29T02:17:27.210 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:17:27.233 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:17:27.233 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:17:27.864 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:17:27.865 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:17:28.899 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:17:28.961 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-29T02:17:28.991 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-29T02:17:29.017 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-29T02:17:29.088 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-29T02:17:29.117 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-29T02:17:29.169 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-29T02:17:29.214 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-29T02:17:29.245 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-29T02:17:29.303 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-29T02:17:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-29T02:17:29.372 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-29T02:17:40.934 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-29T02:17:41.012 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-29T02:17:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-29T02:17:47.199 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-29T02:17:47.199 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-29T02:17:47.199 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-29T02:17:47.634 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-29T02:17:47.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:47.634 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:17:47.634 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-16.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:47.635 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:17:47.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-03-29T02:17:48.513 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:17:48.666 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:17:48.669 INFO:teuthology.orchestra.run.smithi137.stdout:========================================================================================= 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout:========================================================================================= 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 6.5 k 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 2.2 M 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 1.6 M 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 249 k 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 71 M 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T02:17:48.670 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T02:17:48.671 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T02:17:48.672 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout:========================================================================================= 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout:Install 36 Packages 2024-03-29T02:17:48.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:17:48.674 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 84 M 2024-03-29T02:17:48.674 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 921 M 2024-03-29T02:17:48.674 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:17:49.180 INFO:teuthology.orchestra.run.smithi137.stdout:(1/36): ceph-19.0.0-2473.g785fc5e8.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2024-03-29T02:17:50.380 INFO:teuthology.orchestra.run.smithi137.stdout:(2/36): ceph-mgr-19.0.0-2473.g785fc5e8.el9.x86_ 1.1 MB/s | 1.6 MB 00:01 2024-03-29T02:17:50.602 INFO:teuthology.orchestra.run.smithi137.stdout:(3/36): ceph-mds-19.0.0-2473.g785fc5e8.el9.x86_ 1.4 MB/s | 2.2 MB 00:01 2024-03-29T02:17:50.780 INFO:teuthology.orchestra.run.smithi137.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2473.g785f 1.4 MB/s | 249 kB 00:00 2024-03-29T02:17:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T02:17:50.939 INFO:teuthology.orchestra.run.smithi137.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T02:17:50.973 INFO:teuthology.orchestra.run.smithi137.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-29T02:17:51.006 INFO:teuthology.orchestra.run.smithi137.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T02:17:51.040 INFO:teuthology.orchestra.run.smithi137.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T02:17:51.082 INFO:teuthology.orchestra.run.smithi137.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T02:17:51.149 INFO:teuthology.orchestra.run.smithi137.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T02:17:51.183 INFO:teuthology.orchestra.run.smithi137.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T02:17:51.216 INFO:teuthology.orchestra.run.smithi137.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-29T02:17:51.258 INFO:teuthology.orchestra.run.smithi137.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T02:17:51.400 INFO:teuthology.orchestra.run.smithi137.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-03-29T02:17:51.459 INFO:teuthology.orchestra.run.smithi137.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 381 kB/s | 22 kB 00:00 2024-03-29T02:17:51.651 INFO:teuthology.orchestra.run.smithi137.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 789 kB/s | 151 kB 00:00 2024-03-29T02:17:51.712 INFO:teuthology.orchestra.run.smithi137.stdout:(18/36): ceph-mon-19.0.0-2473.g785fc5e8.el9.x86 2.0 MB/s | 5.0 MB 00:02 2024-03-29T02:17:51.737 INFO:teuthology.orchestra.run.smithi137.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 506 kB/s | 43 kB 00:00 2024-03-29T02:17:51.779 INFO:teuthology.orchestra.run.smithi137.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-29T02:17:51.838 INFO:teuthology.orchestra.run.smithi137.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-29T02:17:51.863 INFO:teuthology.orchestra.run.smithi137.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 129 kB/s | 11 kB 00:00 2024-03-29T02:17:51.888 INFO:teuthology.orchestra.run.smithi137.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-29T02:17:51.913 INFO:teuthology.orchestra.run.smithi137.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-29T02:17:51.938 INFO:teuthology.orchestra.run.smithi137.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-29T02:17:51.964 INFO:teuthology.orchestra.run.smithi137.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-29T02:17:51.989 INFO:teuthology.orchestra.run.smithi137.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-03-29T02:17:52.014 INFO:teuthology.orchestra.run.smithi137.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T02:17:52.040 INFO:teuthology.orchestra.run.smithi137.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T02:17:52.065 INFO:teuthology.orchestra.run.smithi137.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-29T02:17:52.090 INFO:teuthology.orchestra.run.smithi137.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T02:17:52.115 INFO:teuthology.orchestra.run.smithi137.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-03-29T02:17:52.141 INFO:teuthology.orchestra.run.smithi137.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T02:17:52.166 INFO:teuthology.orchestra.run.smithi137.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2024-03-29T02:17:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 403 kB/s | 20 kB 00:00 2024-03-29T02:18:01.465 INFO:teuthology.orchestra.run.smithi137.stdout:(36/36): ceph-osd-19.0.0-2473.g785fc5e8.el9.x86 6.4 MB/s | 71 MB 00:11 2024-03-29T02:18:01.471 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:01.471 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.6 MB/s | 84 MB 00:12 2024-03-29T02:18:01.495 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:01.539 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:01.539 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:02.283 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:02.283 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:04.005 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:04.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T02:18:04.110 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T02:18:04.145 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T02:18:04.231 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T02:18:04.269 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T02:18:04.301 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T02:18:04.393 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T02:18:04.513 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T02:18:04.562 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T02:18:04.634 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T02:18:04.688 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T02:18:04.726 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T02:18:04.776 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T02:18:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T02:18:05.014 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T02:18:05.056 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T02:18:05.086 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T02:18:05.121 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T02:18:05.150 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T02:18:05.219 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T02:18:05.253 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T02:18:05.286 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T02:18:05.326 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T02:18:05.361 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T02:18:05.396 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T02:18:05.485 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T02:18:05.535 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T02:18:05.607 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T02:18:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T02:18:05.870 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T02:18:06.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 31/36 2024-03-29T02:18:06.090 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-29T02:18:06.115 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-29T02:18:06.115 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:18:06.115 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T02:18:06.115 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T02:18:06.115 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T02:18:06.116 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:16.965 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-29T02:18:16.993 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-29T02:18:16.993 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:18:16.993 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T02:18:16.993 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T02:18:16.993 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T02:18:16.994 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:17.525 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-29T02:18:17.548 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-29T02:18:17.548 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:18:17.548 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T02:18:17.548 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T02:18:17.548 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T02:18:17.548 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-29T02:18:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-29T02:18:17.816 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:18:17.816 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T02:18:17.816 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T02:18:17.816 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T02:18:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:17.907 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-29T02:18:24.569 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-29T02:18:24.569 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/36 2024-03-29T02:18:24.569 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 6/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T02:18:24.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T02:18:24.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T02:18:24.572 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T02:18:24.572 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T02:18:24.572 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T02:18:24.572 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T02:18:24.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T02:18:24.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T02:18:24.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T02:18:24.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T02:18:24.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T02:18:24.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T02:18:25.564 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T02:18:25.564 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:25.564 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:25.564 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:18:25.564 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:18:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:18:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:18:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:18:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T02:18:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T02:18:25.566 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T02:18:25.567 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:25.568 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:25.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-03-29T02:18:26.415 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:26.513 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:18:26.565 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:26.566 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:18:26.566 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:26.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-03-29T02:18:27.179 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:27.328 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:27.328 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:18:27.328 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:18:27.328 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:18:27.328 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:18:27.328 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 762 k 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 762 k 2024-03-29T02:18:27.329 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 768 k 2024-03-29T02:18:27.330 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:18:27.858 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-19.0.0-2473.g785fc5e8.el9.noarch.rpm 1.4 MB/s | 762 kB 00:00 2024-03-29T02:18:27.859 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:27.859 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.4 MB/s | 762 kB 00:00 2024-03-29T02:18:27.859 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:27.864 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:27.864 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:27.870 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:27.870 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:18:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:18:28.894 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:18:29.208 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:18:29.208 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:29.208 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:29.208 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:29.209 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:29.209 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:29.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T02:18:29.996 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:30.152 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:30.152 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 157 k 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 157 k 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 640 k 2024-03-29T02:18:30.153 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:18:30.497 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-19.0.0-2473.g785fc5 457 kB/s | 157 kB 00:00 2024-03-29T02:18:30.498 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:30.498 INFO:teuthology.orchestra.run.smithi137.stdout:Total 455 kB/s | 157 kB 00:00 2024-03-29T02:18:30.498 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:30.508 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:30.508 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:30.561 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:30.562 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:30.713 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:30.773 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:18:30.797 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:18:30.797 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:18:30.797 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T02:18:30.797 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:31.791 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:18:31.792 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:31.792 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:31.792 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:18:31.792 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:31.792 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:32.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-03-29T02:18:32.580 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:32.676 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:18:32.729 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:32.729 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:18:32.730 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:32.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T02:18:33.349 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:33.498 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:33.499 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-03-29T02:18:33.499 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:18:33.499 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-03-29T02:18:33.499 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:18:33.499 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 3.5 M 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 34 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 16 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:33.500 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:18:33.501 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-03-29T02:18:33.501 INFO:teuthology.orchestra.run.smithi137.stdout:Install 11 Packages 2024-03-29T02:18:33.501 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:33.501 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.6 M 2024-03-29T02:18:33.501 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 100 M 2024-03-29T02:18:33.501 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:18:33.842 INFO:teuthology.orchestra.run.smithi137.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2473.g785 74 kB/s | 16 kB 00:00 2024-03-29T02:18:33.868 INFO:teuthology.orchestra.run.smithi137.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2473.g78 138 kB/s | 34 kB 00:00 2024-03-29T02:18:33.984 INFO:teuthology.orchestra.run.smithi137.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T02:18:34.026 INFO:teuthology.orchestra.run.smithi137.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 6.2 MB/s | 863 kB 00:00 2024-03-29T02:18:34.260 INFO:teuthology.orchestra.run.smithi137.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-29T02:18:34.435 INFO:teuthology.orchestra.run.smithi137.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T02:18:34.544 INFO:teuthology.orchestra.run.smithi137.stdout:(7/11): ceph-mgr-dashboard-19.0.0-2473.g785fc5e 3.8 MB/s | 3.5 MB 00:00 2024-03-29T02:18:34.569 INFO:teuthology.orchestra.run.smithi137.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-29T02:18:34.653 INFO:teuthology.orchestra.run.smithi137.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T02:18:34.720 INFO:teuthology.orchestra.run.smithi137.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.4 MB/s | 2.0 MB 00:00 2024-03-29T02:18:34.787 INFO:teuthology.orchestra.run.smithi137.stdout:(11/11): python3-repoze-lru-0.7-16.el9.noarch.r 127 kB/s | 31 kB 00:00 2024-03-29T02:18:34.791 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:34.791 INFO:teuthology.orchestra.run.smithi137.stdout:Total 5.9 MB/s | 7.6 MB 00:01 2024-03-29T02:18:34.809 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:34.826 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:34.826 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:35.017 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:35.017 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:35.502 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:35.584 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-29T02:18:35.623 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-29T02:18:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-29T02:18:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-29T02:18:35.901 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-29T02:18:35.984 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-29T02:18:36.045 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-29T02:18:36.082 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-29T02:18:36.114 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 9/11 2024-03-29T02:18:37.155 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 10/11 2024-03-29T02:18:37.198 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-29T02:18:38.538 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 1/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 2/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 3/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-29T02:18:38.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-29T02:18:38.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T02:18:38.889 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T02:18:38.890 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T02:18:38.890 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T02:18:38.890 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T02:18:38.890 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T02:18:38.890 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:38.890 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:39.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T02:18:39.709 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:39.861 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:39.862 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-29T02:18:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:18:39.862 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-29T02:18:39.862 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 7.4 M 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T02:18:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:39.864 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 37 M 2024-03-29T02:18:39.865 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 209 M 2024-03-29T02:18:39.865 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:18:40.169 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 240 kB/s | 32 kB 00:00 2024-03-29T02:18:40.203 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-29T02:18:40.245 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 848 kB/s | 34 kB 00:00 2024-03-29T02:18:40.295 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T02:18:40.503 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T02:18:40.570 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T02:18:40.595 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T02:18:40.771 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T02:18:40.804 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 6.2 MB/s | 195 kB 00:00 2024-03-29T02:18:40.972 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-29T02:18:41.213 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 603 kB/s | 245 kB 00:00 2024-03-29T02:18:41.605 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 4.7 MB/s | 7.4 MB 00:01 2024-03-29T02:18:41.608 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:41.608 INFO:teuthology.orchestra.run.smithi137.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T02:18:41.647 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:41.670 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:41.670 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:42.038 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:42.038 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:42.747 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T02:18:42.851 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T02:18:42.944 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T02:18:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T02:18:43.156 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T02:18:43.187 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T02:18:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T02:18:43.920 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T02:18:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T02:18:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T02:18:46.281 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T02:18:46.311 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 1/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T02:18:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T02:18:48.849 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T02:18:48.849 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T02:18:48.849 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T02:18:48.849 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T02:18:48.849 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T02:18:49.259 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T02:18:49.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T02:18:49.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T02:18:49.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T02:18:49.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T02:18:49.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:49.260 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:49.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-03-29T02:18:50.231 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:50.384 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 49 k 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T02:18:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-03-29T02:18:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:50.387 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-03-29T02:18:50.387 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-03-29T02:18:50.387 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:18:50.594 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 18 kB 00:00 2024-03-29T02:18:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 24 kB 00:00 2024-03-29T02:18:50.662 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T02:18:50.686 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T02:18:50.703 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-29T02:18:50.728 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T02:18:50.762 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): ceph-mgr-rook-19.0.0-2473.g785fc5e8.el9 166 kB/s | 49 kB 00:00 2024-03-29T02:18:50.962 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-29T02:18:51.005 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-29T02:18:51.204 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 296 kB/s | 59 kB 00:00 2024-03-29T02:18:51.255 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 412 kB/s | 203 kB 00:00 2024-03-29T02:18:51.380 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-29T02:18:51.514 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T02:18:51.515 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:51.516 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-29T02:18:51.516 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:51.529 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:51.529 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:51.621 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:51.623 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:52.154 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:52.319 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T02:18:52.381 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T02:18:52.437 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T02:18:52.474 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T02:18:52.507 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T02:18:52.601 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T02:18:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T02:18:52.767 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T02:18:53.490 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T02:18:53.547 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T02:18:53.581 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T02:18:53.629 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T02:18:53.642 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 1/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T02:18:54.445 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T02:18:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T02:18:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T02:18:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T02:18:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T02:18:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T02:18:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T02:18:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T02:18:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:54.826 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T02:18:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:54.827 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:55.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T02:18:55.624 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:18:55.779 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:18:55.779 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T02:18:55.779 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 141 k 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-03-29T02:18:55.780 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:55.781 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 6.8 M 2024-03-29T02:18:55.781 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-03-29T02:18:55.781 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:18:56.390 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-29T02:18:56.582 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-29T02:18:56.673 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2473.g785fc5e8.e 344 kB/s | 141 kB 00:00 2024-03-29T02:18:56.807 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 257 kB/s | 58 kB 00:00 2024-03-29T02:18:56.841 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T02:18:56.899 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 226 kB/s | 51 kB 00:00 2024-03-29T02:18:56.903 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:18:56.903 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-29T02:18:56.916 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:18:56.926 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:18:56.926 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:18:57.014 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:18:57.015 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:18:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:18:57.488 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T02:18:57.543 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T02:18:58.052 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T02:18:58.136 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T02:18:58.199 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T02:18:58.238 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-29T02:18:59.120 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-29T02:18:59.121 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/6 2024-03-29T02:18:59.121 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T02:18:59.121 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T02:18:59.121 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T02:18:59.121 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T02:18:59.412 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:18:59.413 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:18:59.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-03-29T02:19:00.216 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:00.366 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 871 k 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:00.367 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:00.368 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:00.368 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-03-29T02:19:00.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:00.368 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 949 k 2024-03-29T02:19:00.368 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 4.0 M 2024-03-29T02:19:00.368 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:00.526 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 498 kB/s | 79 kB 00:00 2024-03-29T02:19:00.702 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-19.0.0-2473.g785fc5e8.el9.x86_ 2.6 MB/s | 871 kB 00:00 2024-03-29T02:19:00.703 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:00.703 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.8 MB/s | 949 kB 00:00 2024-03-29T02:19:00.703 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:00.726 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:00.726 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:00.800 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:00.800 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:01.014 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:01.161 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T02:19:01.227 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-29T02:19:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-29T02:19:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/2 2024-03-29T02:19:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T02:19:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:02.297 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T02:19:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:02.297 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:02.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-03-29T02:19:03.202 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:03.354 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:03.355 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:03.355 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T02:19:03.355 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 264 k 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 332 k 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.5 M 2024-03-29T02:19:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:03.472 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 594 kB/s | 69 kB 00:00 2024-03-29T02:19:03.623 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-volume-19.0.0-2473.g785fc5e8.el9.no 992 kB/s | 264 kB 00:00 2024-03-29T02:19:03.624 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:03.624 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T02:19:03.624 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:03.633 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:03.633 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:03.673 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:03.673 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:03.876 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:04.064 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T02:19:04.109 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-29T02:19:04.124 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-29T02:19:04.124 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:19:04.124 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T02:19:04.125 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 1/2 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:05.173 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:05.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-03-29T02:19:06.036 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:06.193 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:06.194 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:06.194 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T02:19:06.194 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:06.194 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:06.194 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 125 k 2024-03-29T02:19:06.194 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 125 k 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 445 k 2024-03-29T02:19:06.195 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:06.521 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-19.0.0-2473.g785fc5e8.el9.x86_64 383 kB/s | 125 kB 00:00 2024-03-29T02:19:06.522 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:06.522 INFO:teuthology.orchestra.run.smithi137.stdout:Total 381 kB/s | 125 kB 00:00 2024-03-29T02:19:06.522 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:06.527 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:06.527 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:06.583 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:06.583 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:06.724 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:06.836 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:07.364 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:07.688 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:07.689 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:19:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:07.689 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:07.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-03-29T02:19:08.466 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:08.568 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:08.622 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:08.623 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:08.623 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:08.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-03-29T02:19:09.249 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:09.402 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 32 k 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:09.403 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 32 k 2024-03-29T02:19:09.404 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 142 k 2024-03-29T02:19:09.404 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:09.584 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-19.0.0-2473.g785fc5e8.el9.x86_6 175 kB/s | 32 kB 00:00 2024-03-29T02:19:09.585 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:09.585 INFO:teuthology.orchestra.run.smithi137.stdout:Total 174 kB/s | 32 kB 00:00 2024-03-29T02:19:09.585 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:09.588 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:09.588 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:09.609 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:09.609 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:09.699 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:10.253 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:10.509 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:10.509 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:10.509 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:10.510 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:19:10.510 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:10.510 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:10.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-03-29T02:19:11.246 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:11.344 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:11.395 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:11.396 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:11.396 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:11.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-03-29T02:19:12.016 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:12.113 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:12.164 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:12.165 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:12.165 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:12.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-03-29T02:19:12.788 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:12.888 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:12.940 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:12.941 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:12.941 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:13.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-03-29T02:19:13.565 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:13.662 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:13.714 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:13.715 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:13.715 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:13.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-03-29T02:19:14.346 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:14.498 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:14.499 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:14.499 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:14.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-03-29T02:19:15.117 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:15.216 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:19:15.268 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:15.269 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T02:19:15.269 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:15.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-03-29T02:19:15.895 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 90 k 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 90 k 2024-03-29T02:19:16.047 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 313 k 2024-03-29T02:19:16.047 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:16.371 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 277 kB/s | 90 kB 00:00 2024-03-29T02:19:16.371 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:16.371 INFO:teuthology.orchestra.run.smithi137.stdout:Total 275 kB/s | 90 kB 00:00 2024-03-29T02:19:16.372 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:16.380 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:16.380 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:16.436 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:16.436 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:16.585 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:16.694 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:17.144 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:17.436 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:17.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:17.436 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:17.436 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:19:17.437 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:17.437 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:17.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-03-29T02:19:18.260 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:18.417 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:18.418 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:18.418 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T02:19:18.418 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:18.418 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:18.418 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.4 M 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 29 M 2024-03-29T02:19:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:18.823 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 8.4 MB/s | 3.4 MB 00:00 2024-03-29T02:19:18.824 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:18.824 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.4 MB/s | 3.4 MB 00:00 2024-03-29T02:19:18.824 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:18.835 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:18.835 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:18.911 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:18.911 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:19.482 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:19.532 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:19.556 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:19.556 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:19:19.557 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T02:19:19.557 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T02:19:19.557 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T02:19:19.557 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:20.876 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:20.876 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:20.876 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:20.877 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:19:20.877 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:20.877 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:21.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-03-29T02:19:21.708 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 02:16:52 AM UTC. 2024-03-29T02:19:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 176 k 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T02:19:21.859 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T02:19:21.860 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:21.860 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 176 k 2024-03-29T02:19:21.860 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 673 k 2024-03-29T02:19:21.860 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T02:19:22.076 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 812 kB/s | 176 kB 00:00 2024-03-29T02:19:22.077 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:19:22.077 INFO:teuthology.orchestra.run.smithi137.stdout:Total 806 kB/s | 176 kB 00:00 2024-03-29T02:19:22.077 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T02:19:22.087 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T02:19:22.087 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T02:19:22.143 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T02:19:22.143 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T02:19:22.295 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T02:19:22.410 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:22.890 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:23.167 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:19:23.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:23.167 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T02:19:23.167 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:19:23.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:23.167 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T02:19:23.441 DEBUG:teuthology.parallel:result is None 2024-03-29T02:19:23.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:19:23.586 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T02:19:23.606 INFO:teuthology.orchestra.run.smithi137.stdout:19.0.0-2473.g785fc5e8.el9 2024-03-29T02:19:23.607 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2473.g785fc5e8.el9 2024-03-29T02:19:23.607 INFO:teuthology.task.install:The correct ceph version 19.0.0-2473.g785fc5e8 is installed. 2024-03-29T02:19:23.609 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T02:19:23.609 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:23.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T02:19:23.674 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T02:19:23.674 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:23.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T02:19:23.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T02:19:23.808 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T02:19:23.808 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:23.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T02:19:23.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T02:19:23.942 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T02:19:23.942 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:23.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T02:19:24.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T02:19:24.076 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T02:19:24.182 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T02:19:24.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-03-29T02:19:24.210 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T02:19:24.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T02:19:24.277 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T02:19:24.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T02:19:24.352 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T02:19:24.352 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-29T02:19:24.352 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066', 'branch': 'ci-yingxin-crimson-load-obc-3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7629066', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'targets': {'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErUD6NxtA2tbihaaJOJjdayyeu7io1ALB2tFTs1euHgmOHRFVVazHU+TXcng8iY850x09JMjQGUuX0tSuiFLUU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-29_01:54:41', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T02:19:24.353 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T02:19:24.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T02:19:24.467 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:24.467 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T02:19:24.522 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T02:19:24.522 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 02:19:19.842283659 +0000 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 02:16:07.192514189 +0000 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 02:16:07.192514189 +0000 2024-03-29T02:19:24.578 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 02:16:07.192514189 +0000 2024-03-29T02:19:24.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T02:19:24.643 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T02:19:24.643 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T02:19:24.644 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000200346 s, 2.6 MB/s 2024-03-29T02:19:24.644 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T02:19:24.700 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-03-29T02:19:24.755 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 02:19:19.842283659 +0000 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 02:16:07.533507924 +0000 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 02:16:07.533507924 +0000 2024-03-29T02:19:24.756 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 02:16:07.533507924 +0000 2024-03-29T02:19:24.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T02:19:24.821 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T02:19:24.821 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T02:19:24.821 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.0002064 s, 2.5 MB/s 2024-03-29T02:19:24.822 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T02:19:24.878 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-03-29T02:19:24.933 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T02:19:24.934 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:19:24.934 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T02:19:24.934 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:19:24.934 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:19:24.934 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 02:19:19.842283659 +0000 2024-03-29T02:19:24.935 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 02:16:07.862501880 +0000 2024-03-29T02:19:24.935 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 02:16:07.862501880 +0000 2024-03-29T02:19:24.935 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 02:16:07.862501880 +0000 2024-03-29T02:19:24.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T02:19:25.000 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T02:19:25.000 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T02:19:25.000 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000213295 s, 2.4 MB/s 2024-03-29T02:19:25.002 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T02:19:25.057 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-03-29T02:19:25.113 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T02:19:25.113 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:19:25.113 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T02:19:25.113 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:19:25.113 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:19:25.114 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 02:19:19.843283642 +0000 2024-03-29T02:19:25.114 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 02:16:08.213495431 +0000 2024-03-29T02:19:25.114 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 02:16:08.213495431 +0000 2024-03-29T02:19:25.114 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 02:16:08.213495431 +0000 2024-03-29T02:19:25.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T02:19:25.178 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T02:19:25.179 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T02:19:25.179 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000223048 s, 2.3 MB/s 2024-03-29T02:19:25.180 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T02:19:25.235 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:19:25.235 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-29T02:19:25.235 INFO:tasks.ceph:Generating config... 2024-03-29T02:19:25.236 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T02:19:25.236 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T02:19:25.236 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T02:19:25.236 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] log to file = False 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-29T02:19:25.237 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T02:19:25.238 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T02:19:25.238 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T02:19:25.238 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T02:19:25.238 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T02:19:25.238 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] auth client required = none 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] auth cluster required = none 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] auth service required = none 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] auth supported = none 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] debug asok = 0/0 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] debug auth = 0/0 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-03-29T02:19:25.238 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug client = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug context = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug crush = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug filer = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug journal = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-03-29T02:19:25.239 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mds = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug mon = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug monc = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug ms = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-03-29T02:19:25.240 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug osd = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rados = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug refs = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-03-29T02:19:25.241 INFO:tasks.ceph:[global] debug striper = 0/0 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] debug timer = 0/0 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] debug tp = 0/0 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-29T02:19:25.242 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-29T02:19:25.242 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T02:19:25.242 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T02:19:25.242 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-03-29T02:19:25.242 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-03-29T02:19:25.243 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-03-29T02:19:25.243 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-29T02:19:25.243 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-29T02:19:25.243 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-29T02:19:25.243 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-29T02:19:25.243 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-29T02:19:25.243 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T02:19:25.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T02:19:25.339 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T02:19:25.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T02:19:25.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T02:19:25.424 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.137')] 2024-03-29T02:19:25.425 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.137', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-29T02:19:25.426 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:25.426 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T02:19:25.480 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.137 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:25.571 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:25.572+0000 7ffa8bc44f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:25.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:25.572+0000 7ffa8bc44f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:25.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:25.572+0000 7ffa8bc44f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:25.572 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:25.572 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: generated fsid a11d5f07-7396-4e8b-99ad-dbef7ba3ed60 2024-03-29T02:19:25.572 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = quincy 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 0 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:fsid a11d5f07-7396-4e8b-99ad-dbef7ba3ed60 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:last_changed 2024-03-29T02:19:25.573750+0000 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-03-29T02:19:25.573750+0000 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:min_mon_release 17 (quincy) 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:election_strategy: 1 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.a 2024-03-29T02:19:25.573 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-29T02:19:25.574 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T02:19:25.589 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a11d5f07-7396-4e8b-99ad-dbef7ba3ed60... 2024-03-29T02:19:25.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = none 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.137 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: auth client required = none 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: auth cluster required = none 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: auth service required = none 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: debug asok = 0/0 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 0/0 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: debug bdev = 0/0 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 0/0 2024-03-29T02:19:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug buffer = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug civetweb = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug compressor = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug context = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug crush = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug crypto = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug dpdk = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug eventtrace = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug filer = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug filestore = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug finisher = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug fuse = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug heartbeatmap = 0/0 2024-03-29T02:19:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: debug javaclient = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug journal = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug journaler = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug kstore = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug leveldb = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug lockdep = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds balancer = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds locker = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds log = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds log expire = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds migrator = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug memdb = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 0/0 2024-03-29T02:19:25.685 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgrc = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug monc = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug objclass = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug objectcacher = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug objecter = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug optracker = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug perfcounter = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug rados = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug rbd = 0/0 2024-03-29T02:19:25.686 INFO:teuthology.orchestra.run.smithi137.stdout: debug rbd mirror = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug rbd replay = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug refs = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw lifecycle = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw notification = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw sync = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug striper = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug throttle = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug timer = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: debug tp = 0/0 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: ms client mode = crc 2024-03-29T02:19:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: ms cluster mode = crc 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: ms mon client mode = crc 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: ms mon cluster mode = crc 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: ms mon service mode = crc 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: ms service mode = crc 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = a11d5f07-7396-4e8b-99ad-dbef7ba3ed60 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-03-29T02:19:25.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-03-29T02:19:25.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: debug monc = 20 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore block size = 96636764160 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore compression algorithm = snappy 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore compression mode = aggressive 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore fsck on mount = True 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore rocksdb cf = False 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: crimson alien thread cpu cores = 6-7 2024-03-29T02:19:25.690 INFO:teuthology.orchestra.run.smithi137.stdout: crimson osd obc lru size = 10 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 20 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 20 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug filestore = 0/0 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug journal = 0/0 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 0/0 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 0/0 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 10 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: err to stderr = True 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: log flush on exit = True 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: log to file = False 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: log to stderr = True 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd full ratio = 0.9 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd nearfull ratio = 0.8 2024-03-29T02:19:25.691 INFO:teuthology.orchestra.run.smithi137.stdout: osd failsafe full ratio = 0.95 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = bluestore 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: osd shutdown pgref assert = True 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-03-29T02:19:25.692 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 0/0 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 0/0 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 0/0 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T02:19:25.693 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default crimson = True 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-03-29T02:19:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 0-2 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 3-5 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 0-2 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.3] 2024-03-29T02:19:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 3-5 2024-03-29T02:19:25.696 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T02:19:25.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T02:19:25.760 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T02:19:25.761 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:25.761 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T02:19:25.775 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:25.775 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T02:19:25.829 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:19:25.829 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:25.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T02:19:25.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T02:19:25.908 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:25.909 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:25.964 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T02:19:25.964 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T02:19:25.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T02:19:26.066 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T02:19:26.069 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T02:19:26.070 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T02:19:26.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T02:19:26.120 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T02:19:26.136 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T02:19:26.136 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-29T02:19:26.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T02:19:26.163 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:19:26.163 INFO:tasks.ceph:role: osd.0 2024-03-29T02:19:26.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:19:26.163 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:19:26.237 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:19:26.238 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:19:26.238 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:19:26.238 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:19:26.753 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T02:19:26.756 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T02:19:26.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T02:19:26.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T02:19:26.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T02:19:27.007 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:19:27.007 INFO:tasks.ceph:role: osd.1 2024-03-29T02:19:27.007 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:19:27.007 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:19:27.082 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:19:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:19:27.083 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:19:27.529 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T02:19:27.531 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T02:19:27.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T02:19:27.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T02:19:27.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T02:19:27.724 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:19:27.724 INFO:tasks.ceph:role: osd.2 2024-03-29T02:19:27.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:19:27.724 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T02:19:27.798 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:19:27.798 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:19:27.798 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:19:27.799 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:19:28.302 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T02:19:28.304 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T02:19:28.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T02:19:28.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T02:19:28.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:28.769 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:19:28.770 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-03-29 02:19:28,772 seastar - Reactor backend: linux-aio 2024-03-29T02:19:28.780 INFO:teuthology.orchestra.run.smithi137.stderr:==29084==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:19:28.915 INFO:teuthology.orchestra.run.smithi137.stderr:DEBUG 2024-03-29 02:19:28,917 [shard 0:main] seastar - reactor::drain 2024-03-29T02:19:29.456 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:29.456 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-03-29T02:19:29.456 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-03-29T02:19:29.456 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-03-29T02:19:29.456 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:29.456 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:19:29.514 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:19:29.518 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-03-29 02:19:29,520 seastar - Reactor backend: linux-aio 2024-03-29T02:19:29.527 INFO:teuthology.orchestra.run.smithi137.stderr:==29082==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:19:29.619 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x172f7 0x174bb 0x2eab 0x4a09d 0x35b7c 0x3629e 0x5d8bd74 0x5d8bf70 0x5d8c16d 0x5d8c443 0x5ddac83 0x5ddae89 0x5d59970 0x5d59b4a 0x5d5a0d6 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x362658a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:19:29.619 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-03-29T02:19:34.175 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:34.176 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-03-29T02:19:34.176 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-03-29T02:19:34.176 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-03-29T02:19:34.176 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:34.176 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:19:34.176 INFO:teuthology.orchestra.run.smithi137.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid a11d5f07-7396-4e8b-99ad-dbef7ba3ed60 2024-03-29T02:19:34.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T02:19:34.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:34.514 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:19:34.514 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-03-29 02:19:34,516 seastar - Reactor backend: linux-aio 2024-03-29T02:19:34.519 INFO:teuthology.orchestra.run.smithi137.stderr:==29266==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:19:34.651 INFO:teuthology.orchestra.run.smithi137.stderr:DEBUG 2024-03-29 02:19:34,652 [shard 0:main] seastar - reactor::drain 2024-03-29T02:19:35.184 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:35.184 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-03-29T02:19:35.184 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-03-29T02:19:35.184 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-03-29T02:19:35.185 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:35.185 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:19:35.242 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:19:35.246 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-03-29 02:19:35,248 seastar - Reactor backend: linux-aio 2024-03-29T02:19:35.266 INFO:teuthology.orchestra.run.smithi137.stderr:==29264==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:19:39.943 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:39.943 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-03-29T02:19:39.943 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-03-29T02:19:39.944 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-03-29T02:19:39.944 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:39.944 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:19:39.944 INFO:teuthology.orchestra.run.smithi137.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid a11d5f07-7396-4e8b-99ad-dbef7ba3ed60 2024-03-29T02:19:39.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T02:19:39.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:40.277 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:19:40.278 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-03-29 02:19:40,280 seastar - Reactor backend: linux-aio 2024-03-29T02:19:40.282 INFO:teuthology.orchestra.run.smithi137.stderr:==29452==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:19:40.389 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x362fa 0x378ece2 0x42cba8a 0x4371c5b 0x5d7d55d 0x5d7d78e 0x5d59f25 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x3785266 0x3785dbf 0x3625d7a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:19:40.389 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-03-29T02:19:40.414 INFO:teuthology.orchestra.run.smithi137.stderr:DEBUG 2024-03-29 02:19:40,416 [shard 0:main] seastar - reactor::drain 2024-03-29T02:19:40.948 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:40.948 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-03-29T02:19:40.948 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-03-29T02:19:40.949 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-03-29T02:19:40.949 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:40.949 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:19:41.001 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:19:41.005 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-03-29 02:19:41,006 seastar - Reactor backend: linux-aio 2024-03-29T02:19:41.017 INFO:teuthology.orchestra.run.smithi137.stderr:==29450==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:19:45.660 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:45.660 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-03-29T02:19:45.661 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-03-29T02:19:45.661 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-03-29T02:19:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-03-29T02:19:45.661 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:19:45.662 INFO:teuthology.orchestra.run.smithi137.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid a11d5f07-7396-4e8b-99ad-dbef7ba3ed60 2024-03-29T02:19:45.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T02:19:45.708 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T02:19:45.708 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:45.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T02:19:45.775 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:45.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T02:19:45.841 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:45.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T02:19:45.909 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:45.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T02:19:45.976 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:45.976 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T02:19:46.031 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T02:19:46.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T02:19:46.097 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-03-29T02:19:46.097 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAuJQZmC5sbBBAAMd5Rjs3uxEe5FprXVuzjvg== 2024-03-29T02:19:46.097 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-03-29T02:19:46.097 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAxJQZm79VzKBAAYKctjzozHjW8ounl9pIg8w== 2024-03-29T02:19:46.098 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-03-29T02:19:46.098 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA3JQZmYlfPGBAALY1qIGx5NnYQO4vD4lDJuA== 2024-03-29T02:19:46.098 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-03-29T02:19:46.098 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA9JQZmXKkZChAAon3YZqHMbjUDYTeAtdT4UA== 2024-03-29T02:19:46.098 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-03-29T02:19:46.098 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAuJQZmZCpOBxAAf609lw2hyJIH1dj8gs3vEA== 2024-03-29T02:19:46.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T02:19:46.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:19:46.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:19:46.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:19:46.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T02:19:46.452 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T02:19:46.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T02:19:46.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T02:19:46.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:46.599+0000 7f4d87caaa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:46.599+0000 7f4d87caaa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T02:19:46.642 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:19:46.696 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T02:19:46.697 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T02:19:46.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T02:19:46.739 INFO:tasks.ceph.mon.a:Started 2024-03-29T02:19:46.739 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T02:19:46.739 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T02:19:46.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T02:19:46.742 INFO:tasks.ceph.mgr.x:Started 2024-03-29T02:19:46.742 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T02:19:46.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T02:19:46.824 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-03-29T02:19:46.825+0000 7f2e3ae3ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.824 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-03-29T02:19:46.825+0000 7f2e3ae3ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.824 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-03-29T02:19:46.825+0000 7f2e3ae3ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.826 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:46.827+0000 7f35c822c000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:46.851+0000 7f7465824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:46.858+0000 7f7465824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:46.925 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-03-29T02:19:46.936 INFO:tasks.ceph:check_enable_crimson: True 2024-03-29T02:19:46.937 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-29T02:19:46.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-29T02:19:47.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:47.031+0000 7f35c822c000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:47.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:47.031+0000 7f35c822c000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:47.047 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:47.043+0000 7f9f7062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:47.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:47.050+0000 7f9f7062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:47.126 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-03-29T02:19:47.127+0000 7f2e317f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-29T02:19:47.127 INFO:tasks.ceph.mon.a.smithi137.stderr:Please be aware that this feature is experimental, untested, 2024-03-29T02:19:47.127 INFO:tasks.ceph.mon.a.smithi137.stderr:unsupported, and may result in data corruption, data loss, 2024-03-29T02:19:47.127 INFO:tasks.ceph.mon.a.smithi137.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-29T02:19:47.127 INFO:tasks.ceph.mon.a.smithi137.stderr:feature with important data. 2024-03-29T02:19:47.127 INFO:tasks.ceph.mon.a.smithi137.stderr: 2024-03-29T02:19:47.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:47.500+0000 7f35c822c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T02:19:47.553 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:47.553+0000 7f35c822c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T02:19:47.937 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T02:19:47.938 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:47.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T02:19:47.964 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:47.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T02:19:48.032 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T02:19:48.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T02:19:48.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.075+0000 7f35c822c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T02:19:48.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 349961b1-28d0-4f4c-8dd2-3a7bfad66874 0 2024-03-29T02:19:48.131 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.132+0000 7f35c822c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T02:19:48.178 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.179+0000 7f35c822c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T02:19:48.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:48.237+0000 7f1b0202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:48.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:48.245+0000 7f1b0202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:48.255 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.256+0000 7f35c822c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T02:19:48.298 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.299+0000 7f35c822c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T02:19:48.320 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-03-29T02:19:48.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 610bf83e-b516-4933-b991-755f153366f5 1 2024-03-29T02:19:48.423 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.423+0000 7f35c822c000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T02:19:48.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:48.437+0000 7f803bdc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:48.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:48.445+0000 7f803bdc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:48.494 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.495+0000 7f35c822c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T02:19:48.520 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-03-29T02:19:48.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new c094eddd-8fbd-43ef-b4b3-59dac901b481 2 2024-03-29T02:19:48.543 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.544+0000 7f35c822c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T02:19:48.599 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.600+0000 7f35c822c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T02:19:48.641 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:48.636+0000 7feb9466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:48.643 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:48.644+0000 7feb9466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:48.649 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:48.650+0000 7f35c822c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T02:19:48.721 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-03-29T02:19:48.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-29T02:19:48.805 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T02:19:48.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-29T02:19:48.848 INFO:tasks.ceph.osd.0:Started 2024-03-29T02:19:48.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-29T02:19:48.880 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T02:19:48.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-29T02:19:48.922 INFO:tasks.ceph.osd.1:Started 2024-03-29T02:19:48.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-29T02:19:48.956 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T02:19:48.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-29T02:19:48.998 INFO:tasks.ceph.osd.2:Started 2024-03-29T02:19:48.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:19:49.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:49.069+0000 7f35c822c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T02:19:49.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:49.121+0000 7f891db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:49.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T02:19:49.128+0000 7f891db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:49.198 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T02:19:49.198 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":6,"fsid":"a11d5f07-7396-4e8b-99ad-dbef7ba3ed60","created":"2024-03-29T02:19:46.843647+0000","modified":"2024-03-29T02:19:48.722154+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T02:19:48.722154+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"349961b1-28d0-4f4c-8dd2-3a7bfad66874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"610bf83e-b516-4933-b991-755f153366f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c094eddd-8fbd-43ef-b4b3-59dac901b481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:19:49.211 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T02:19:49.211 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T02:19:49.300 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:49.301+0000 7f35c822c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T02:19:49.493 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:49.494+0000 7f35c822c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T02:19:49.513 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:19:49.574 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:49.575+0000 7f35c822c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T02:19:49.614 INFO:teuthology.misc.health.smithi137.stderr:2024-03-29T02:19:49.612+0000 7fabf16ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:49.616 INFO:teuthology.misc.health.smithi137.stderr:2024-03-29T02:19:49.617+0000 7fabf16ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:49.682 INFO:teuthology.misc.health.smithi137.stdout: 2024-03-29T02:19:49.683 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":6,"fsid":"a11d5f07-7396-4e8b-99ad-dbef7ba3ed60","created":"2024-03-29T02:19:46.843647+0000","modified":"2024-03-29T02:19:48.722154+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T02:19:48.722154+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"349961b1-28d0-4f4c-8dd2-3a7bfad66874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"610bf83e-b516-4933-b991-755f153366f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c094eddd-8fbd-43ef-b4b3-59dac901b481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:19:49.693 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-29T02:19:49.786 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:49.787+0000 7f35c822c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T02:19:49.950 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:49.951+0000 7f35c822c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T02:19:50.222 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:50.222+0000 7f35c822c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T02:19:50.292 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:50.293+0000 7f35c822c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T02:19:50.568 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:50.569+0000 7f35c822c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T02:19:51.833 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:51.834+0000 7f35c822c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T02:19:52.295 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:52.296+0000 7f35c822c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T02:19:52.381 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:52.382+0000 7f35c822c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T02:19:52.542 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:52.542+0000 7f35c822c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T02:19:53.551 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T02:19:53.552+0000 7f3566f30640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-29T02:19:55.995 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:19:56.101 INFO:teuthology.misc.health.smithi137.stderr:2024-03-29T02:19:56.098+0000 7f1eb646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:56.102 INFO:teuthology.misc.health.smithi137.stderr:2024-03-29T02:19:56.104+0000 7f1eb646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:19:56.293 INFO:teuthology.misc.health.smithi137.stdout: 2024-03-29T02:19:56.293 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":8,"fsid":"a11d5f07-7396-4e8b-99ad-dbef7ba3ed60","created":"2024-03-29T02:19:46.843647+0000","modified":"2024-03-29T02:19:55.553143+0000","last_up_change":"2024-03-29T02:19:55.553143+0000","last_in_change":"2024-03-29T02:19:48.722154+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"349961b1-28d0-4f4c-8dd2-3a7bfad66874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6809","nonce":3336621361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3336621361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3336621361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":3336621361}]},"public_addr":"172.21.15.137:6809/3336621361","cluster_addr":"172.21.15.137:6808/3336621361","heartbeat_back_addr":"172.21.15.137:6806/3336621361","heartbeat_front_addr":"172.21.15.137:6803/3336621361","state":["exists","up"]},{"osd":1,"uuid":"610bf83e-b516-4933-b991-755f153366f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6805","nonce":1918409588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":1918409588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1918409588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6815","nonce":1918409588}]},"public_addr":"172.21.15.137:6805/1918409588","cluster_addr":"172.21.15.137:6804/1918409588","heartbeat_back_addr":"172.21.15.137:6816/1918409588","heartbeat_front_addr":"172.21.15.137:6815/1918409588","state":["exists","up"]},{"osd":2,"uuid":"c094eddd-8fbd-43ef-b4b3-59dac901b481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:19:56.304 DEBUG:teuthology.misc:2 of 3 OSDs are up 2024-03-29T02:20:02.406 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 864, in wait_until_osds_up daemon.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-03-29T02:20:02.408 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T02:20:02.408 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T02:20:02.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:20:02.458 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T02:20:02.459 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T02:20:02.459 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:20:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:20:02.459 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 864, in wait_until_osds_up daemon.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-03-29T02:20:02.459 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T02:20:02.459 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T02:20:02.460 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:20:02.515 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T02:20:02.515 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T02:20:02.515 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T02:20:02.515 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:20:02.542 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T02:20:02.542 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T02:20:02.543 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T02:20:02.543 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:20:02.593 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T02:20:02.594 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T02:20:02.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-29T02:20:02.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:20:02.665 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T02:20:02.977 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:20:02.977 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T02:20:03.099 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:20:03.099 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T02:20:03.191 INFO:tasks.ceph:Archiving mon data... 2024-03-29T02:20:03.192 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066/data/mon.a.tgz 2024-03-29T02:20:03.192 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-03-29T02:20:03.206 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.p4nAndXCcj 2024-03-29T02:20:03.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.p4nAndXCcj 2024-03-29T02:20:03.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.p4nAndXCcj 2024-03-29T02:20:03.395 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.p4nAndXCcj is 126KB 2024-03-29T02:20:03.447 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.p4nAndXCcj 2024-03-29T02:20:03.461 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T02:20:03.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T02:20:03.584 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T02:20:03.584 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T02:20:03.586 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066/remote/smithi137/crash 2024-03-29T02:20:03.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T02:20:03.650 INFO:tasks.ceph:Compressing logs... 2024-03-29T02:20:03.650 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T02:20:03.722 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30409.log 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29973.log 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30409.log.gz 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr: 89.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29973.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28598.log 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29973.log.gz 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30844.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T02:20:03.723 INFO:teuthology.orchestra.run.smithi137.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.tmp-client.admin.28598.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28598.log.gz 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30061.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30439.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr: 85.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30171.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30061.log.gz 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30439.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30439.log.gz 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30211.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30171.log.gz 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30251.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30211.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T02:20:03.724 INFO:teuthology.orchestra.run.smithi137.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.30211.log.gz 2024-03-29T02:20:03.725 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30251.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T02:20:03.725 INFO:teuthology.orchestra.run.smithi137.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30251.log.gz 2024-03-29T02:20:03.725 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T02:20:03.725 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.log: 76.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T02:20:03.727 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.2.log: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T02:20:03.728 INFO:teuthology.orchestra.run.smithi137.stderr: 90.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T02:20:03.737 INFO:teuthology.orchestra.run.smithi137.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T02:20:03.738 INFO:teuthology.orchestra.run.smithi137.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T02:20:03.739 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T02:20:03.740 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m0.035s 2024-03-29T02:20:03.740 INFO:teuthology.orchestra.run.smithi137.stderr:user 0m0.064s 2024-03-29T02:20:03.740 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m0.023s 2024-03-29T02:20:03.740 INFO:tasks.ceph:Archiving logs... 2024-03-29T02:20:03.740 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066/remote/smithi137/log 2024-03-29T02:20:03.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T02:20:03.870 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 864, in wait_until_osds_up daemon.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-03-29T02:20:04.075 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2018263de6b44c8d8397fc9a3c6ca30a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 864, in wait_until_osds_up daemon.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-03-29T02:20:04.079 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T02:20:04.088 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T02:20:04.096 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T02:20:04.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T02:20:04.154 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T02:20:04.163 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T02:20:04.163 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T02:20:04.182 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-03-29T02:20:04.187 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T02:20:04.187 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-03-29T02:20:04.187 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 2 6 177 14 +785us[ +755us] +/- 75ms 2024-03-29T02:20:04.187 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 6 177 14 -31us[ -31us] +/- 53ms 2024-03-29T02:20:04.187 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 177 14 -855us[ -855us] +/- 83ms 2024-03-29T02:20:04.187 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:20:04.188 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T02:20:04.196 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T02:20:04.197 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T02:20:04.205 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T02:20:04.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T02:20:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:20:04.319 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T02:20:04.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T02:20:04.337 INFO:teuthology.task.internal:Duration was 466.831038 seconds 2024-03-29T02:20:04.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T02:20:04.345 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T02:20:04.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T02:20:04.402 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T02:20:04.740 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T02:20:04.741 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T02:20:04.741 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T02:20:04.801 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T02:20:04.801 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T02:20:04.871 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T02:20:04.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T02:20:05.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T02:20:05.015 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T02:20:05.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T02:20:05.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T02:20:05.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T02:20:05.108 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-03-29T02:20:05.135 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T02:20:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:20:05.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T02:20:05.201 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T02:20:05.202 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066/remote/smithi137 2024-03-29T02:20:05.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T02:20:05.267 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T02:20:05.267 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T02:20:05.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T02:20:05.326 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T02:20:05.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T02:20:05.335 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T02:20:05.335 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T02:20:05.370 INFO:teuthology.orchestra.run.smithi137.stdout: 265317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 02:20 /home/ubuntu/cephtest 2024-03-29T02:20:05.371 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T02:20:05.383 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T02:20:05.406 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T02:20:05.427 DEBUG:teuthology.nuke:shortname: smithi137 2024-03-29T02:20:05.428 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T02:20:05.446 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 02:04:13.731424', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErUD6NxtA2tbihaaJOJjdayyeu7io1ALB2tFTs1euHgmOHRFVVazHU+TXcng8iY850x09JMjQGUuX0tSuiFLUU='} 2024-03-29T02:20:05.493 INFO:teuthology.orchestra.console.smithi137:Power off 2024-03-29T02:20:05.493 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T02:20:05.517 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-03-29T02:20:05.518 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T02:20:05.541 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-03-29T02:20:09.542 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T02:20:09.651 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-03-29T02:20:13.654 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T02:20:13.764 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-03-29T02:20:13.865 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-03-29T02:20:13.970 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 466.8310384750366 failure_reason: 'Command failed on smithi137 with status 1: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2018263de6b44c8d8397fc9a3c6ca30a status: fail success: false 2024-03-29T02:20:13.970 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T02:20:14.059 INFO:teuthology.run:FAIL