2023-10-17T06:58:45.009 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T06:58:45.010 DEBUG:teuthology.run:Teuthology command: teuthology --name sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi --description crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} --verbose --archive /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067 --owner scheduled_sjust@teuthology -- /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067/orig.config.yaml 2023-10-17T06:58:45.038 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T06:58:45.135 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067 branch: wip-crimson-scrub-testing-2023-10-16 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} email: null first_in_suite: false job_id: '7431067' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-crimson-scrub-testing-2023-10-16 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson seastar smp: 3 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(POOL_FULL\) sha1: a83d484198747052d1ed9eeed0a72e20b007e411 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: a83d484198747052d1ed9eeed0a72e20b007e411 workunit: branch: wip-crimson-scrub-testing-2023-10-16 env: CRIMSON_COMPAT: '1' sha1: a83d484198747052d1ed9eeed0a72e20b007e411 owner: scheduled_sjust@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3210 sha1: a83d484198747052d1ed9eeed0a72e20b007e411 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-crimson-scrub-testing-2023-10-16 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a83d484198747052d1ed9eeed0a72e20b007e411 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkX6Uym4JX933EQyRfA93ZVWZvAPT32MnwBP+A/EbyOtGI3nWlbNM/lVWzrn3EPzD/zsLqup9dsaX5G55qcH1Y= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh env: RBD_FEATURES: '61' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-17_06:50:25 tube: smithi user: sjust verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-17T06:58:45.136 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa; will attempt to use it 2023-10-17T06:58:45.136 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks 2023-10-17T06:58:45.137 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-17T06:58:45.139 INFO:teuthology.task.internal:Checking packages... 2023-10-17T06:58:45.162 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'a83d484198747052d1ed9eeed0a72e20b007e411' 2023-10-17T06:58:45.163 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-17T06:58:45.163 INFO:teuthology.packaging:ref: None 2023-10-17T06:58:45.163 INFO:teuthology.packaging:tag: None 2023-10-17T06:58:45.164 INFO:teuthology.packaging:branch: wip-crimson-scrub-testing-2023-10-16 2023-10-17T06:58:45.164 INFO:teuthology.packaging:sha1: a83d484198747052d1ed9eeed0a72e20b007e411 2023-10-17T06:58:45.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-crimson-scrub-testing-2023-10-16 2023-10-17T06:58:45.311 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6790.ga83d4841 2023-10-17T06:58:45.313 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-17T06:58:45.321 INFO:teuthology.task.internal:no buildpackages task found 2023-10-17T06:58:45.322 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-17T06:58:45.335 INFO:teuthology.task.internal:Saving configuration 2023-10-17T06:58:45.347 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-17T06:58:45.357 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T06:58:45.383 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 06:52:06.191836', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkX6Uym4JX933EQyRfA93ZVWZvAPT32MnwBP+A/EbyOtGI3nWlbNM/lVWzrn3EPzD/zsLqup9dsaX5G55qcH1Y='} 2023-10-17T06:58:45.384 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-17T06:58:45.390 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-17T06:58:45.390 INFO:teuthology.run_tasks:Running task console_log... 2023-10-17T06:58:45.437 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1e0ed27940>, signals=[15]) 2023-10-17T06:58:45.438 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-17T06:58:45.448 INFO:teuthology.task.internal:Opening connections... 2023-10-17T06:58:45.449 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T06:58:45.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T06:58:45.551 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-17T06:58:45.559 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2023-10-17T06:58:45.624 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2023-10-17T06:58:45.625 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2023-10-17T06:58:45.682 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2023-10-17T06:58:45.682 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2023-10-17T06:58:45.682 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2023-10-17T06:58:45.682 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2023-10-17T06:58:45.683 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2023-10-17T06:58:45.683 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2023-10-17T06:58:45.683 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-17T06:58:45.683 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2023-10-17T06:58:45.683 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2023-10-17T06:58:45.684 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-17T06:58:45.684 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2023-10-17T06:58:45.684 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-17T06:58:45.684 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-17T06:58:45.684 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-17T06:58:45.685 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2023-10-17T06:58:45.715 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-17T06:58:45.727 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-17T06:58:45.733 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-17T06:58:45.734 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2023-10-17T06:58:45.751 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-17T06:58:45.758 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-17T06:58:45.758 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2023-10-17T06:58:45.819 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-17T06:58:45.854 INFO:teuthology.run_tasks:Running task kernel... 2023-10-17T06:58:45.871 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-17T06:58:45.873 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-17T06:58:45.873 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-17T06:58:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T06:58:45.892 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-10-17T06:58:45.949 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-361.el9.x86_64 2023-10-17T06:58:45.950 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-361.el9.x86_64 2023-10-17T06:58:45.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2023-10-17T06:58:48.029 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-17T06:58:49.422 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2023-10-17T06:58:51.412 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 7.9 MB 00:01 2023-10-17T06:58:54.380 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 18 MB 00:01 2023-10-17T06:58:59.780 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.7 MB 00:01 2023-10-17T06:59:01.189 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 14 kB 00:00 2023-10-17T06:59:03.009 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-10-17T06:59:07.648 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-10-17T06:59:09.681 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-17T06:59:09.681 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-17T06:59:09.769 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T06:59:09.778 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T06:59:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T06:59:09.779 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T06:59:09.779 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T06:59:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-375.el9 CentOS-BaseOS 5.3 M 2023-10-17T06:59:09.779 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T06:59:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-375.el9 CentOS-BaseOS 20 M 2023-10-17T06:59:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-375.el9 CentOS-BaseOS 37 M 2023-10-17T06:59:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-375.el9 CentOS-BaseOS 32 M 2023-10-17T06:59:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T06:59:09.780 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T06:59:09.780 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T06:59:09.781 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2023-10-17T06:59:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T06:59:09.781 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 95 M 2023-10-17T06:59:09.782 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 120 M 2023-10-17T06:59:09.782 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T06:59:10.125 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-375.el9.x86_64.rpm 15 MB/s | 5.3 MB 00:00 2023-10-17T06:59:10.575 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-375.el9.x86_64.rpm 25 MB/s | 20 MB 00:00 2023-10-17T06:59:11.150 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-5.14.0-375.el9.x86_64.rpm 27 MB/s | 37 MB 00:01 2023-10-17T06:59:11.642 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-core-5.14.0-375.el9.x86_6 21 MB/s | 32 MB 00:01 2023-10-17T06:59:11.646 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T06:59:11.646 INFO:teuthology.orchestra.run.smithi049.stdout:Total 51 MB/s | 95 MB 00:01 2023-10-17T06:59:11.646 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T06:59:12.087 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T06:59:12.087 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T06:59:13.123 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T06:59:13.124 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T06:59:15.650 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T06:59:16.316 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-375.el9.x86_64 1/4 2023-10-17T06:59:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-17T06:59:19.327 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-17T06:59:19.504 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-17T06:59:24.470 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-17T06:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-375.el9.x86_64 4/4 2023-10-17T06:59:29.559 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-17T06:59:59.974 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 4/4 2023-10-17T07:00:00.019 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 4/4 2023-10-17T07:00:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-375.el9.x86_64 4/4 2023-10-17T07:00:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-375.el9.x86_64 1/4 2023-10-17T07:00:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-17T07:00:00.515 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-17T07:00:00.993 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-17T07:00:00.993 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:00:00.993 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:00:00.993 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-375.el9.x86_64 2023-10-17T07:00:00.994 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-375.el9.x86_64 2023-10-17T07:00:00.994 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-375.el9.x86_64 2023-10-17T07:00:00.994 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-375.el9.x86_64 2023-10-17T07:00:00.994 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:00:00.994 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:00:01.492 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-10-17T07:00:01.624 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-17T07:00:01.625 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-17T07:00:01.626 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-375.el9.x86_64 2023-10-17T07:00:01.626 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-17T07:00:01.626 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-17T07:00:01.626 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-17T07:00:01.627 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2023-10-17T07:00:01.627 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-10-17T07:00:01.736 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-17T07:00:01.736 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-17T07:00:01.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2023-10-17T07:00:01.774 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2023-10-17T07:00:01.775 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2023-10-17T07:00:01.775 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2023-10-17T07:00:01.775 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 68.el9 2023-10-17T07:00:01.775 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2023-10-17T07:00:01.775 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-17T07:00:01.776 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2023-10-17T07:00:01.776 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8264254 2023-10-17T07:00:01.776 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2023-10-17T07:00:01.776 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-17T07:00:01.776 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-17T07:00:01.777 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-17T07:00:01.777 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-17T07:00:01.777 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2023-10-17T07:00:01.777 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2023-10-17T07:00:01.777 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-17T07:00:01.777 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2023-10-17T07:00:01.778 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2023-10-17T07:00:01.778 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:00:01.778 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-17T07:00:01.778 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-17T07:00:01.778 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-17T07:00:01.779 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2023-10-17T07:00:01.779 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:00:01.779 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2023-10-17T07:00:01.780 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-17T07:00:01.780 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-375.el9.x86_64 2023-10-17T07:00:01.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-17T07:00:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2023-10-17T07:00:03.771 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-17T07:00:03.806 INFO:teuthology.orchestra.run.smithi049.stderr:done 2023-10-17T07:00:03.808 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-10-17T07:00:03.825 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.hhAHmfc4VC 2023-10-17T07:00:03.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hhAHmfc4VC 2023-10-17T07:00:03.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.hhAHmfc4VC 2023-10-17T07:00:04.044 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.hhAHmfc4VC is 6KB 2023-10-17T07:00:04.092 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.hhAHmfc4VC 2023-10-17T07:00:04.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2023-10-17T07:00:04.174 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-17T07:00:04.175 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-17T07:00:04.175 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-17T07:00:04.175 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64.conf 2023-10-17T07:00:04.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64 2023-10-17T07:00:04.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2023-10-17T07:00:34.350 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-17T07:00:34.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:00:34.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T07:01:08.859 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-10-17T07:01:17.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:01:17.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T07:01:20.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-10-17T07:01:32.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:01:32.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T07:01:35.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-10-17T07:01:51.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:01:51.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T07:02:51.071 DEBUG:teuthology.orchestra.remote:timed out 2023-10-17T07:03:09.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:03:09.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T07:03:09.398 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-10-17T07:03:09.786 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:03:09.786 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-375.el9.x86_64"... 2023-10-17T07:03:09.787 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-10-17T07:03:09.804 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-375.el9.x86_64 2023-10-17T07:03:09.804 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-375.el9.x86_64 vs 5.14.0-375.el9.x86_64 2023-10-17T07:03:09.804 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-17T07:03:09.804 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-17T07:03:10.805 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-17T07:03:10.806 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-17T07:03:10.896 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2023-10-17T07:03:10.932 DEBUG:teuthology.parallel:result is None 2023-10-17T07:03:10.933 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-17T07:03:10.940 INFO:teuthology.task.internal:Creating test directory... 2023-10-17T07:03:10.940 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-17T07:03:10.958 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-17T07:03:10.965 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-17T07:03:10.975 INFO:teuthology.task.internal:Creating archive directory... 2023-10-17T07:03:10.976 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-17T07:03:11.026 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-17T07:03:11.033 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-17T07:03:11.034 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-17T07:03:11.107 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T07:03:11.124 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T07:03:11.126 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-17T07:03:11.135 INFO:teuthology.task.internal:Configuring sudo... 2023-10-17T07:03:11.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-17T07:03:11.169 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-17T07:03:11.178 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-17T07:03:11.179 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-17T07:03:11.225 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T07:03:11.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T07:03:11.362 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T07:03:11.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T07:03:11.491 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:03:11.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-17T07:03:11.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2023-10-17T07:03:11.648 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T07:03:11.767 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-17T07:03:11.776 INFO:teuthology.task.internal:Starting timer... 2023-10-17T07:03:11.776 INFO:teuthology.run_tasks:Running task pcp... 2023-10-17T07:03:11.786 INFO:teuthology.run_tasks:Running task selinux... 2023-10-17T07:03:11.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2023-10-17T07:03:11.863 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2023-10-17T07:03:11.865 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-17T07:03:11.865 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2023-10-17T07:03:11.898 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2023-10-17T07:03:11.898 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive'} 2023-10-17T07:03:11.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T07:03:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T07:03:11.970 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-17T07:03:11.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2023-10-17T07:03:12.050 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-17T07:03:12.061 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-17T07:03:12.067 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-17T07:03:12.187 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-17T07:03:12.203 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-17T07:03:12.205 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-17T07:08:46.468 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2023-10-17T07:08:46.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:08:46.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T07:08:46.572 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-10-17T07:08:46.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T07:08:46.641 INFO:teuthology.run_tasks:Running task clock... 2023-10-17T07:08:46.652 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-17T07:08:46.653 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-17T07:08:46.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T07:08:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-17T07:08:46.738 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-17T07:08:46.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2023-10-17T07:08:46.792 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2023-10-17T07:08:46.812 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-17T07:08:46.833 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-17T07:08:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2023-10-17T07:08:46.885 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T07:08:46.885 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-10-17T07:08:46.885 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T07:08:46.885 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T07:08:46.885 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T07:08:46.886 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T07:08:46.887 INFO:teuthology.run_tasks:Running task install... 2023-10-17T07:08:46.895 DEBUG:teuthology.task.install:project ceph 2023-10-17T07:08:46.896 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411'}} 2023-10-17T07:08:46.896 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411'} 2023-10-17T07:08:46.896 INFO:teuthology.task.install:Using flavor: crimson 2023-10-17T07:08:46.902 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-17T07:08:46.902 INFO:teuthology.task.install:extra packages: [] 2023-10-17T07:08:46.903 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411', 'tag': None, 'wait_for_package': False} 2023-10-17T07:08:46.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a83d484198747052d1ed9eeed0a72e20b007e411 2023-10-17T07:08:47.119 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-16/a83d484198747052d1ed9eeed0a72e20b007e411/centos/9/flavors/crimson/ 2023-10-17T07:08:47.119 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6790.ga83d4841 2023-10-17T07:08:47.239 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-16/a83d484198747052d1ed9eeed0a72e20b007e411/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-16/a83d484198747052d1ed9eeed0a72e20b007e411/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-16/a83d484198747052d1ed9eeed0a72e20b007e411/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-17T07:08:47.240 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:08:47.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-17T07:08:47.271 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-17T07:08:47.271 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a83d484198747052d1ed9eeed0a72e20b007e411/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-17T07:08:47.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-17T07:08:47.430 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-17T07:08:47.461 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2023-10-17T07:08:47.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2023-10-17T07:08:47.710 INFO:teuthology.orchestra.run.smithi049.stdout:72 files removed 2023-10-17T07:08:47.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2023-10-17T07:08:48.460 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 209 kB/s | 81 kB 00:00 2023-10-17T07:08:48.802 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-10-17T07:08:49.170 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2023-10-17T07:08:49.547 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-10-17T07:08:50.947 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2023-10-17T07:08:52.891 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.9 MB 00:01 2023-10-17T07:08:56.808 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 18 MB 00:02 2023-10-17T07:09:03.006 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.7 MB 00:02 2023-10-17T07:09:04.507 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 14 kB 00:00 2023-10-17T07:09:07.953 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 19 MB 00:03 2023-10-17T07:09:12.508 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2023-10-17T07:09:14.914 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:09:14.928 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:09:14.929 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:09:14.929 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:09:14.929 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:09:14.930 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 8.4 M 2023-10-17T07:09:14.930 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2023-10-17T07:09:14.930 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 3.1 M 2023-10-17T07:09:14.931 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 3.0 M 2023-10-17T07:09:14.931 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:09:14.932 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options 2023-10-17T07:09:14.932 INFO:teuthology.orchestra.run.smithi049.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-17T07:09:14.932 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 6.6 M 2023-10-17T07:09:14.932 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 17 M 2023-10-17T07:09:14.932 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 24 k 2023-10-17T07:09:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-17T07:09:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-17T07:09:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-17T07:09:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-17T07:09:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 727 k 2023-10-17T07:09:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-17T07:09:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-17T07:09:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-17T07:09:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 469 k 2023-10-17T07:09:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-17T07:09:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 4.5 M 2023-10-17T07:09:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-17T07:09:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-17T07:09:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-17T07:09:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse 2023-10-17T07:09:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 45 k 2023-10-17T07:09:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 127 k 2023-10-17T07:09:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 162 k 2023-10-17T07:09:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-17T07:09:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 321 k 2023-10-17T07:09:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 297 k 2023-10-17T07:09:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 99 k 2023-10-17T07:09:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-17T07:09:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-17T07:09:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:14.937 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:09:14.938 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:09:14.938 INFO:teuthology.orchestra.run.smithi049.stdout:Install 28 Packages 2023-10-17T07:09:14.938 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2023-10-17T07:09:14.938 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:14.939 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 54 M 2023-10-17T07:09:14.939 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:09:15.884 INFO:teuthology.orchestra.run.smithi049.stdout:(1/30): ceph-base-18.0.0-6790.ga83d4841.el9.x86 8.0 MB/s | 6.6 MB 00:00 2023-10-17T07:09:15.918 INFO:teuthology.orchestra.run.smithi049.stdout:(2/30): ceph-selinux-18.0.0-6790.ga83d4841.el9. 725 kB/s | 24 kB 00:00 2023-10-17T07:09:16.001 INFO:teuthology.orchestra.run.smithi049.stdout:(3/30): libcephfs2-18.0.0-6790.ga83d4841.el9.x8 8.5 MB/s | 727 kB 00:00 2023-10-17T07:09:16.068 INFO:teuthology.orchestra.run.smithi049.stdout:(4/30): libradosstriper1-18.0.0-6790.ga83d4841. 6.9 MB/s | 469 kB 00:00 2023-10-17T07:09:16.452 INFO:teuthology.orchestra.run.smithi049.stdout:(5/30): librgw2-18.0.0-6790.ga83d4841.el9.x86_6 12 MB/s | 4.5 MB 00:00 2023-10-17T07:09:16.494 INFO:teuthology.orchestra.run.smithi049.stdout:(6/30): python3-ceph-argparse-18.0.0-6790.ga83d 1.1 MB/s | 45 kB 00:00 2023-10-17T07:09:16.544 INFO:teuthology.orchestra.run.smithi049.stdout:(7/30): python3-ceph-common-18.0.0-6790.ga83d48 2.5 MB/s | 127 kB 00:00 2023-10-17T07:09:16.602 INFO:teuthology.orchestra.run.smithi049.stdout:(8/30): python3-cephfs-18.0.0-6790.ga83d4841.el 2.7 MB/s | 162 kB 00:00 2023-10-17T07:09:16.669 INFO:teuthology.orchestra.run.smithi049.stdout:(9/30): python3-rados-18.0.0-6790.ga83d4841.el9 4.7 MB/s | 321 kB 00:00 2023-10-17T07:09:16.786 INFO:teuthology.orchestra.run.smithi049.stdout:(10/30): ceph-radosgw-18.0.0-6790.ga83d4841.el9 4.9 MB/s | 8.4 MB 00:01 2023-10-17T07:09:16.811 INFO:teuthology.orchestra.run.smithi049.stdout:(11/30): python3-rbd-18.0.0-6790.ga83d4841.el9. 2.0 MB/s | 297 kB 00:00 2023-10-17T07:09:16.870 INFO:teuthology.orchestra.run.smithi049.stdout:(12/30): python3-rgw-18.0.0-6790.ga83d4841.el9. 1.2 MB/s | 99 kB 00:00 2023-10-17T07:09:16.937 INFO:teuthology.orchestra.run.smithi049.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 844 kB/s | 106 kB 00:00 2023-10-17T07:09:16.971 INFO:teuthology.orchestra.run.smithi049.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2023-10-17T07:09:17.021 INFO:teuthology.orchestra.run.smithi049.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2023-10-17T07:09:17.046 INFO:teuthology.orchestra.run.smithi049.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-10-17T07:09:17.080 INFO:teuthology.orchestra.run.smithi049.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 710 kB/s | 41 kB 00:00 2023-10-17T07:09:17.180 INFO:teuthology.orchestra.run.smithi049.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 337 kB/s | 45 kB 00:00 2023-10-17T07:09:17.397 INFO:teuthology.orchestra.run.smithi049.stdout:(19/30): ceph-common-18.0.0-6790.ga83d4841.el9. 7.4 MB/s | 17 MB 00:02 2023-10-17T07:09:17.464 INFO:teuthology.orchestra.run.smithi049.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 288 kB/s | 111 kB 00:00 2023-10-17T07:09:17.590 INFO:teuthology.orchestra.run.smithi049.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 755 kB/s | 309 kB 00:00 2023-10-17T07:09:17.631 INFO:teuthology.orchestra.run.smithi049.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 146 kB/s | 24 kB 00:00 2023-10-17T07:09:17.798 INFO:teuthology.orchestra.run.smithi049.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2023-10-17T07:09:17.865 INFO:teuthology.orchestra.run.smithi049.stdout:(24/30): libarrow-9.0.0-9.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2023-10-17T07:09:17.890 INFO:teuthology.orchestra.run.smithi049.stdout:(25/30): libunwind-1.6.2-1.el9.x86_64.rpm 260 kB/s | 67 kB 00:00 2023-10-17T07:09:18.107 INFO:teuthology.orchestra.run.smithi049.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 793 kB/s | 192 kB 00:00 2023-10-17T07:09:18.374 INFO:teuthology.orchestra.run.smithi049.stdout:(27/30): librados2-18.0.0-6790.ga83d4841.el9.x8 12 MB/s | 3.1 MB 00:00 2023-10-17T07:09:18.491 INFO:teuthology.orchestra.run.smithi049.stdout:(28/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.2 MB/s | 837 kB 00:00 2023-10-17T07:09:18.566 INFO:teuthology.orchestra.run.smithi049.stdout:(29/30): librbd1-18.0.0-6790.ga83d4841.el9.x86_ 16 MB/s | 3.0 MB 00:00 2023-10-17T07:09:18.892 INFO:teuthology.orchestra.run.smithi049.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:01 2023-10-17T07:09:18.896 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:09:18.896 INFO:teuthology.orchestra.run.smithi049.stdout:Total 14 MB/s | 54 MB 00:03 2023-10-17T07:09:18.909 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:09:18.951 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:09:18.951 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:09:19.445 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:09:19.446 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:09:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:09:21.238 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-17T07:09:21.302 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:18.0.0-6790.ga83d4841.el9.x86_64 2/32 2023-10-17T07:09:21.403 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:18.0.0-6790.ga83d4841.el9.x86_64 2/32 2023-10-17T07:09:21.494 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:18.0.0-6790.ga83d4841.el9.x86_64 3/32 2023-10-17T07:09:21.520 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:18.0.0-6790.ga83d4841.el9.x86_64 4/32 2023-10-17T07:09:21.583 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:18.0.0-6790.ga83d4841.el9.x86_64 4/32 2023-10-17T07:09:21.621 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-17T07:09:21.650 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-17T07:09:21.679 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-17T07:09:21.723 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:18.0.0-6790.ga83d4841.el9. 8/32 2023-10-17T07:09:21.792 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:18.0.0-6790.ga83d4841.el9.x86_64 9/32 2023-10-17T07:09:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:18.0.0-6790.ga83d4841.el9.x86_6 10/32 2023-10-17T07:09:21.904 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6790.ga83d4841.el9.x86_6 10/32 2023-10-17T07:09:21.938 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-17T07:09:22.049 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-17T07:09:22.084 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-17T07:09:22.146 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-17T07:09:22.182 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-17T07:09:22.209 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-17T07:09:22.245 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-17T07:09:22.278 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-17T07:09:22.606 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-17T07:09:22.632 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:18.0.0-6790.ga83d4841.el9.x86_64 20/32 2023-10-17T07:09:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:18.0.0-6790.ga83d4841.el9.x86_64 20/32 2023-10-17T07:09:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:18.0.0-6790.ga83d4841.el9.x86_64 21/32 2023-10-17T07:09:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-17T07:09:23.244 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-17T07:09:23.268 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:18.0.0-6790.ga83d4841.el9.x86_64 24/32 2023-10-17T07:09:23.339 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:18.0.0-6790.ga83d4841.el9.x86_64 24/32 2023-10-17T07:09:23.420 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:18.0.0-6790.ga83d4841.el9.x86_64 25/32 2023-10-17T07:09:23.462 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:18.0.0-6790.ga83d4841.el9.x8 26/32 2023-10-17T07:09:25.602 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-6790.ga83d4841.el9.x86_64 27/32 2023-10-17T07:09:25.634 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:18.0.0-6790.ga83d4841.el9.x86_64 27/32 2023-10-17T07:09:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-6790.ga83d4841.el9.x86_64 27/32 2023-10-17T07:09:26.568 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:18.0.0-6790.ga83d4841.el9.x86_64 28/32 2023-10-17T07:09:26.669 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:18.0.0-6790.ga83d4841.el9.x86_64 28/32 2023-10-17T07:09:26.669 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-17T07:09:26.669 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-17T07:09:26.670 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:26.705 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:18.0.0-6790.ga83d4841.el9.x86_64 29/32 2023-10-17T07:09:36.595 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6790.ga83d4841.el9.x86_64 29/32 2023-10-17T07:09:36.596 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2023-10-17T07:09:36.596 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2023-10-17T07:09:36.596 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2023-10-17T07:09:36.596 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2023-10-17T07:09:36.596 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2023-10-17T07:09:36.597 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2023-10-17T07:09:36.597 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2023-10-17T07:09:36.597 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:37.454 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:18.0.0-6790.ga83d4841.el9.x86_64 30/32 2023-10-17T07:09:37.479 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6790.ga83d4841.el9.x86_64 30/32 2023-10-17T07:09:37.480 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:09:37.480 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-17T07:09:37.480 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-17T07:09:37.481 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-17T07:09:37.481 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:37.482 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-17T07:09:37.574 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-17T07:09:37.574 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-17T07:09:38.712 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-17T07:09:38.712 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:18.0.0-6790.ga83d4841.el9.x86_64 1/32 2023-10-17T07:09:38.713 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:18.0.0-6790.ga83d4841.el9.x86_64 2/32 2023-10-17T07:09:38.713 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:18.0.0-6790.ga83d4841.el9.x86_64 3/32 2023-10-17T07:09:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:18.0.0-6790.ga83d4841.el9.x86_64 4/32 2023-10-17T07:09:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:18.0.0-6790.ga83d4841.el9.x86_64 5/32 2023-10-17T07:09:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:18.0.0-6790.ga83d4841.el9.x86_6 6/32 2023-10-17T07:09:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:18.0.0-6790.ga83d4841.el9.x86_64 7/32 2023-10-17T07:09:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6790.ga83d4841.el9. 8/32 2023-10-17T07:09:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:18.0.0-6790.ga83d4841.el9.x8 9/32 2023-10-17T07:09:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:18.0.0-6790.ga83d4841.el9.x86_64 10/32 2023-10-17T07:09:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:18.0.0-6790.ga83d4841.el9.x86_64 11/32 2023-10-17T07:09:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:18.0.0-6790.ga83d4841.el9.x86_64 12/32 2023-10-17T07:09:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:18.0.0-6790.ga83d4841.el9.x86_64 13/32 2023-10-17T07:09:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-17T07:09:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-17T07:09:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-17T07:09:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-17T07:09:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-17T07:09:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-17T07:09:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-17T07:09:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-17T07:09:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-17T07:09:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-17T07:09:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-17T07:09:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-17T07:09:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-17T07:09:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-17T07:09:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-17T07:09:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:18.0.0-6790.ga83d4841.el9.x86_64 29/32 2023-10-17T07:09:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-17T07:09:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:18.0.0-6790.ga83d4841.el9.x86_64 31/32 2023-10-17T07:09:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-17T07:09:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:39.632 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2023-10-17T07:09:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.632 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:09:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-17T07:09:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-17T07:09:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-17T07:09:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-17T07:09:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-17T07:09:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-17T07:09:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-17T07:09:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-17T07:09:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-17T07:09:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-17T07:09:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-17T07:09:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-17T07:09:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-17T07:09:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:09:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2023-10-17T07:09:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-17T07:09:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:39.638 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:09:40.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2023-10-17T07:09:40.639 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:09:40.786 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:09:40.787 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:09:40.787 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:09:40.788 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:09:40.788 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:09:40.788 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 117 M 2023-10-17T07:09:40.788 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:09:40.788 INFO:teuthology.orchestra.run.smithi049.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-17T07:09:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-17T07:09:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-17T07:09:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 161 k 2023-10-17T07:09:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-17T07:09:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-17T07:09:40.790 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-17T07:09:40.790 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-17T07:09:40.790 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-17T07:09:40.790 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:40.790 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:09:40.791 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:09:40.791 INFO:teuthology.orchestra.run.smithi049.stdout:Install 10 Packages 2023-10-17T07:09:40.791 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:09:40.792 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 120 M 2023-10-17T07:09:40.792 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.1 G 2023-10-17T07:09:40.792 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:09:41.187 INFO:teuthology.orchestra.run.smithi049.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.3 MB/s | 418 kB 00:00 2023-10-17T07:09:41.229 INFO:teuthology.orchestra.run.smithi049.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-10-17T07:09:41.271 INFO:teuthology.orchestra.run.smithi049.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-10-17T07:09:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-10-17T07:09:41.346 INFO:teuthology.orchestra.run.smithi049.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-17T07:09:41.413 INFO:teuthology.orchestra.run.smithi049.stdout:(6/10): libcephsqlite-18.0.0-6790.ga83d4841.el9 403 kB/s | 161 kB 00:00 2023-10-17T07:09:41.497 INFO:teuthology.orchestra.run.smithi049.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-10-17T07:09:41.956 INFO:teuthology.orchestra.run.smithi049.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 266 kB/s | 122 kB 00:00 2023-10-17T07:09:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.9 MB/s | 1.3 MB 00:00 2023-10-17T07:09:46.148 INFO:teuthology.orchestra.run.smithi049.stdout:(10/10): ceph-test-18.0.0-6790.ga83d4841.el9.x8 23 MB/s | 117 MB 00:05 2023-10-17T07:09:46.150 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:09:46.150 INFO:teuthology.orchestra.run.smithi049.stdout:Total 22 MB/s | 120 MB 00:05 2023-10-17T07:09:46.165 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:09:46.190 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:09:46.190 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:09:47.044 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:09:47.045 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:09:48.196 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:09:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-17T07:09:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-17T07:09:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-17T07:09:48.418 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-17T07:09:48.448 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-17T07:09:48.479 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-17T07:09:48.508 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-17T07:09:48.551 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-17T07:09:48.577 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:18.0.0-6790.ga83d4841.el9.x86_64 9/10 2023-10-17T07:10:05.821 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6790.ga83d4841.el9.x86_64 9/10 2023-10-17T07:10:05.902 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:18.0.0-6790.ga83d4841.el9.x86_64 10/10 2023-10-17T07:10:13.907 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:18.0.0-6790.ga83d4841.el9.x86_64 10/10 2023-10-17T07:10:13.907 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:18.0.0-6790.ga83d4841.el9.x86_64 1/10 2023-10-17T07:10:13.908 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:18.0.0-6790.ga83d4841.el9.x86_64 2/10 2023-10-17T07:10:13.908 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-17T07:10:13.908 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-17T07:10:13.909 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-17T07:10:13.909 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-17T07:10:13.909 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-17T07:10:13.909 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-17T07:10:13.909 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-17T07:10:14.330 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-17T07:10:14.331 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:14.331 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:10:14.331 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:14.331 INFO:teuthology.orchestra.run.smithi049.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-17T07:10:14.332 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-15.el9.x86_64 2023-10-17T07:10:14.332 INFO:teuthology.orchestra.run.smithi049.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-17T07:10:14.332 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:14.332 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-17T07:10:14.333 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-17T07:10:14.333 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-17T07:10:14.333 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-17T07:10:14.333 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-17T07:10:14.333 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:14.334 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:10:14.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2023-10-17T07:10:15.241 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:03 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:10:15.393 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:10:15.395 INFO:teuthology.orchestra.run.smithi049.stdout:========================================================================================= 2023-10-17T07:10:15.396 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:10:15.396 INFO:teuthology.orchestra.run.smithi049.stdout:========================================================================================= 2023-10-17T07:10:15.396 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:10:15.396 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 6.5 k 2023-10-17T07:10:15.396 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:10:15.397 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 2.0 M 2023-10-17T07:10:15.397 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 1.5 M 2023-10-17T07:10:15.397 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 242 k 2023-10-17T07:10:15.397 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 8.8 M 2023-10-17T07:10:15.397 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 118 M 2023-10-17T07:10:15.398 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-17T07:10:15.398 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-17T07:10:15.398 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-17T07:10:15.398 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-17T07:10:15.398 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-17T07:10:15.399 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-17T07:10:15.399 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-17T07:10:15.399 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-17T07:10:15.399 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-17T07:10:15.399 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-17T07:10:15.400 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-17T07:10:15.400 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-17T07:10:15.400 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-17T07:10:15.400 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-17T07:10:15.400 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-17T07:10:15.400 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-17T07:10:15.401 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-17T07:10:15.401 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-17T07:10:15.401 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-17T07:10:15.401 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-17T07:10:15.401 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-17T07:10:15.402 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-17T07:10:15.402 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-17T07:10:15.402 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-17T07:10:15.402 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-17T07:10:15.402 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-17T07:10:15.403 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-17T07:10:15.403 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:15.403 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:10:15.403 INFO:teuthology.orchestra.run.smithi049.stdout:========================================================================================= 2023-10-17T07:10:15.403 INFO:teuthology.orchestra.run.smithi049.stdout:Install 33 Packages 2023-10-17T07:10:15.404 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:15.404 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 135 M 2023-10-17T07:10:15.404 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.4 G 2023-10-17T07:10:15.405 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:10:16.191 INFO:teuthology.orchestra.run.smithi049.stdout:(1/33): ceph-18.0.0-6790.ga83d4841.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2023-10-17T07:10:16.266 INFO:teuthology.orchestra.run.smithi049.stdout:(2/33): ceph-mds-18.0.0-6790.ga83d4841.el9.x86_ 5.8 MB/s | 2.0 MB 00:00 2023-10-17T07:10:16.300 INFO:teuthology.orchestra.run.smithi049.stdout:(3/33): ceph-mgr-18.0.0-6790.ga83d4841.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2023-10-17T07:10:16.400 INFO:teuthology.orchestra.run.smithi049.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6790.ga83d 2.4 MB/s | 242 kB 00:00 2023-10-17T07:10:16.550 INFO:teuthology.orchestra.run.smithi049.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2023-10-17T07:10:16.584 INFO:teuthology.orchestra.run.smithi049.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2023-10-17T07:10:16.651 INFO:teuthology.orchestra.run.smithi049.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 579 kB/s | 39 kB 00:00 2023-10-17T07:10:16.709 INFO:teuthology.orchestra.run.smithi049.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-10-17T07:10:16.743 INFO:teuthology.orchestra.run.smithi049.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-17T07:10:16.776 INFO:teuthology.orchestra.run.smithi049.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-10-17T07:10:16.810 INFO:teuthology.orchestra.run.smithi049.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-17T07:10:16.852 INFO:teuthology.orchestra.run.smithi049.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-10-17T07:10:17.069 INFO:teuthology.orchestra.run.smithi049.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 5.7 MB/s | 1.2 MB 00:00 2023-10-17T07:10:17.419 INFO:teuthology.orchestra.run.smithi049.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 146 kB/s | 51 kB 00:00 2023-10-17T07:10:17.544 INFO:teuthology.orchestra.run.smithi049.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 349 kB/s | 43 kB 00:00 2023-10-17T07:10:17.603 INFO:teuthology.orchestra.run.smithi049.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2023-10-17T07:10:17.695 INFO:teuthology.orchestra.run.smithi049.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2023-10-17T07:10:17.729 INFO:teuthology.orchestra.run.smithi049.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 321 kB/s | 11 kB 00:00 2023-10-17T07:10:17.762 INFO:teuthology.orchestra.run.smithi049.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 532 kB/s | 18 kB 00:00 2023-10-17T07:10:17.796 INFO:teuthology.orchestra.run.smithi049.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 694 kB/s | 23 kB 00:00 2023-10-17T07:10:17.829 INFO:teuthology.orchestra.run.smithi049.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 583 kB/s | 19 kB 00:00 2023-10-17T07:10:17.863 INFO:teuthology.orchestra.run.smithi049.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2023-10-17T07:10:17.897 INFO:teuthology.orchestra.run.smithi049.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2023-10-17T07:10:17.939 INFO:teuthology.orchestra.run.smithi049.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2023-10-17T07:10:18.006 INFO:teuthology.orchestra.run.smithi049.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2023-10-17T07:10:18.048 INFO:teuthology.orchestra.run.smithi049.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2023-10-17T07:10:18.106 INFO:teuthology.orchestra.run.smithi049.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-10-17T07:10:18.157 INFO:teuthology.orchestra.run.smithi049.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2023-10-17T07:10:18.265 INFO:teuthology.orchestra.run.smithi049.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2023-10-17T07:10:18.349 INFO:teuthology.orchestra.run.smithi049.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2023-10-17T07:10:18.383 INFO:teuthology.orchestra.run.smithi049.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 602 kB/s | 20 kB 00:00 2023-10-17T07:10:19.307 INFO:teuthology.orchestra.run.smithi049.stdout:(32/33): ceph-mon-18.0.0-6790.ga83d4841.el9.x86 2.8 MB/s | 8.8 MB 00:03 2023-10-17T07:10:21.733 INFO:teuthology.orchestra.run.smithi049.stdout:(33/33): ceph-osd-18.0.0-6790.ga83d4841.el9.x86 22 MB/s | 118 MB 00:05 2023-10-17T07:10:21.737 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:10:21.738 INFO:teuthology.orchestra.run.smithi049.stdout:Total 21 MB/s | 135 MB 00:06 2023-10-17T07:10:21.772 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:10:21.815 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:10:21.815 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:10:22.853 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:10:22.853 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:10:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:10:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-17T07:10:24.801 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-17T07:10:24.834 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-17T07:10:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-17T07:10:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-17T07:10:25.071 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-17T07:10:25.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-17T07:10:25.187 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-17T07:10:25.239 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-10-17T07:10:25.276 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-17T07:10:25.326 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-17T07:10:25.389 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-17T07:10:25.587 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-17T07:10:25.631 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-17T07:10:25.669 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-17T07:10:25.704 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-17T07:10:25.733 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-17T07:10:25.805 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-17T07:10:25.866 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-17T07:10:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-17T07:10:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-17T07:10:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-17T07:10:26.073 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-17T07:10:26.109 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-17T07:10:26.186 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-17T07:10:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-17T07:10:26.460 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-17T07:10:26.638 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6790.ga83d4841.el9. 28/33 2023-10-17T07:10:26.664 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:18.0.0-6790.ga83d4841.el9.x86_64 29/33 2023-10-17T07:10:26.691 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6790.ga83d4841.el9.x86_64 29/33 2023-10-17T07:10:26.691 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:10:26.691 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-17T07:10:26.691 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-17T07:10:26.692 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-17T07:10:26.692 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:18.0.0-6790.ga83d4841.el9.x86_64 30/33 2023-10-17T07:10:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:18.0.0-6790.ga83d4841.el9.x86_64 30/33 2023-10-17T07:10:46.324 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:10:46.324 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-17T07:10:46.324 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-17T07:10:46.324 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-17T07:10:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:47.272 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:18.0.0-6790.ga83d4841.el9.x86_64 31/33 2023-10-17T07:10:47.296 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:18.0.0-6790.ga83d4841.el9.x86_64 31/33 2023-10-17T07:10:47.296 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:10:47.296 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-17T07:10:47.296 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-17T07:10:47.297 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-17T07:10:47.297 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:47.506 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:18.0.0-6790.ga83d4841.el9.x86_64 32/33 2023-10-17T07:10:47.529 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:18.0.0-6790.ga83d4841.el9.x86_64 32/33 2023-10-17T07:10:47.529 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:10:47.530 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-17T07:10:47.530 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-17T07:10:47.530 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-17T07:10:47.530 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:47.598 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:18.0.0-6790.ga83d4841.el9.x86_64 33/33 2023-10-17T07:10:57.502 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:18.0.0-6790.ga83d4841.el9.x86_64 33/33 2023-10-17T07:10:57.502 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:18.0.0-6790.ga83d4841.el9.x86_64 1/33 2023-10-17T07:10:57.503 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:18.0.0-6790.ga83d4841.el9.x86_64 2/33 2023-10-17T07:10:57.503 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:18.0.0-6790.ga83d4841.el9.x86_64 3/33 2023-10-17T07:10:57.503 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:18.0.0-6790.ga83d4841.el9.x86_64 4/33 2023-10-17T07:10:57.503 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:18.0.0-6790.ga83d4841.el9.x86_64 5/33 2023-10-17T07:10:57.503 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6790.ga83d4841.el9. 6/33 2023-10-17T07:10:57.504 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-17T07:10:57.504 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-17T07:10:57.504 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-17T07:10:57.504 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-17T07:10:57.504 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-17T07:10:57.505 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-17T07:10:57.505 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-17T07:10:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-10-17T07:10:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-17T07:10:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-17T07:10:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-17T07:10:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-17T07:10:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-17T07:10:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-17T07:10:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-17T07:10:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-17T07:10:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-17T07:10:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-17T07:10:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-17T07:10:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-17T07:10:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-17T07:10:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-17T07:10:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-17T07:10:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-17T07:10:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-17T07:10:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-17T07:10:58.333 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-17T07:10:58.333 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:58.333 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:10:58.334 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:58.334 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:58.334 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:58.334 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:10:58.335 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:58.335 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:10:58.335 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-17T07:10:58.335 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-17T07:10:58.335 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-17T07:10:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-17T07:10:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-17T07:10:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-17T07:10:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-17T07:10:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-17T07:10:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-17T07:10:58.337 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-17T07:10:58.337 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-17T07:10:58.337 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-17T07:10:58.337 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-17T07:10:58.337 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-17T07:10:58.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-17T07:10:58.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2023-10-17T07:10:58.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-17T07:10:58.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-17T07:10:58.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-17T07:10:58.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-17T07:10:58.339 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-17T07:10:58.339 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-17T07:10:58.339 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-17T07:10:58.339 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-17T07:10:58.339 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-17T07:10:58.339 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-17T07:10:58.340 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-17T07:10:58.340 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:10:58.340 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:10:58.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2023-10-17T07:10:59.181 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:10:59.273 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:10:59.325 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:10:59.326 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:10:59.326 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:10:59.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2023-10-17T07:10:59.995 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:00.140 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:00.140 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 270 k 2023-10-17T07:11:00.141 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:00.141 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 270 k 2023-10-17T07:11:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 265 k 2023-10-17T07:11:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:00.594 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-18.0.0-6790.ga83d4841.el9.noarch.rpm 594 kB/s | 270 kB 00:00 2023-10-17T07:11:00.595 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:00.596 INFO:teuthology.orchestra.run.smithi049.stdout:Total 592 kB/s | 270 kB 00:00 2023-10-17T07:11:00.596 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:00.601 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:00.601 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:00.604 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:00.605 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:00.674 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:01.125 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:01.918 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:01.919 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:01.919 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:01.919 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:01.919 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:01.919 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:02.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2023-10-17T07:11:02.736 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:02.880 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:02.881 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:02.881 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T07:11:02.881 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:02.881 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 144 k 2023-10-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:02.883 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:02.883 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 144 k 2023-10-17T07:11:02.883 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 438 k 2023-10-17T07:11:02.883 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:03.216 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-18.0.0-6790.ga83d48 431 kB/s | 144 kB 00:00 2023-10-17T07:11:03.217 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:03.217 INFO:teuthology.orchestra.run.smithi049.stdout:Total 429 kB/s | 144 kB 00:00 2023-10-17T07:11:03.217 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:03.225 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:03.225 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:03.280 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:03.280 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:03.428 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:03.484 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6790.ga83d4841. 1/1 2023-10-17T07:11:03.509 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6790.ga83d4841. 1/1 2023-10-17T07:11:03.510 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:11:03.510 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-17T07:11:03.510 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:04.556 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6790.ga83d4841. 1/1 2023-10-17T07:11:04.556 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:04.557 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:04.557 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:11:04.557 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:04.558 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:04.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2023-10-17T07:11:05.405 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:53 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:05.497 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:05.548 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:05.549 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:05.549 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:05.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2023-10-17T07:11:06.218 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:06.364 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:06.366 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:06.366 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:06.367 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:06.367 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:06.367 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 1.8 M 2023-10-17T07:11:06.367 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:11:06.368 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 23 k 2023-10-17T07:11:06.368 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 15 k 2023-10-17T07:11:06.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-17T07:11:06.369 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-17T07:11:06.369 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-17T07:11:06.369 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2023-10-17T07:11:06.369 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-17T07:11:06.369 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:06.370 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:06.370 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:06.370 INFO:teuthology.orchestra.run.smithi049.stdout:Install 7 Packages 2023-10-17T07:11:06.370 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:06.371 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.1 M 2023-10-17T07:11:06.371 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 8.6 M 2023-10-17T07:11:06.371 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:07.153 INFO:teuthology.orchestra.run.smithi049.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6790.ga83 56 kB/s | 23 kB 00:00 2023-10-17T07:11:07.687 INFO:teuthology.orchestra.run.smithi049.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 17 kB/s | 9.0 kB 00:00 2023-10-17T07:11:07.954 INFO:teuthology.orchestra.run.smithi049.stdout:(3/7): ceph-prometheus-alerts-18.0.0-6790.ga83d 12 kB/s | 15 kB 00:01 2023-10-17T07:11:08.296 INFO:teuthology.orchestra.run.smithi049.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 67 kB/s | 41 kB 00:00 2023-10-17T07:11:08.504 INFO:teuthology.orchestra.run.smithi049.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 56 kB/s | 31 kB 00:00 2023-10-17T07:11:08.921 INFO:teuthology.orchestra.run.smithi049.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 302 kB/s | 188 kB 00:00 2023-10-17T07:11:10.013 INFO:teuthology.orchestra.run.smithi049.stdout:(7/7): ceph-mgr-dashboard-18.0.0-6790.ga83d4841 551 kB/s | 1.8 MB 00:03 2023-10-17T07:11:10.017 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:10.018 INFO:teuthology.orchestra.run.smithi049.stdout:Total 579 kB/s | 2.1 MB 00:03 2023-10-17T07:11:10.018 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:10.028 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:10.028 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:10.102 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:10.103 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-17T07:11:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-17T07:11:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-17T07:11:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-17T07:11:10.622 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6790.ga83d4841.el9.n 5/7 2023-10-17T07:11:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6790.ga83d4841.el9. 6/7 2023-10-17T07:11:10.900 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6790.ga83d4841.el9.noarc 7/7 2023-10-17T07:11:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6790.ga83d4841.el9.noarc 7/7 2023-10-17T07:11:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6790.ga83d4841.el9. 1/7 2023-10-17T07:11:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6790.ga83d4841.el9.noarc 2/7 2023-10-17T07:11:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6790.ga83d4841.el9.n 3/7 2023-10-17T07:11:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-17T07:11:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-17T07:11:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-17T07:11:11.785 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-17T07:11:11.785 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:11.785 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:11.785 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:11.786 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:11.786 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:11.786 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-17T07:11:11.786 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-17T07:11:11.786 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-17T07:11:11.787 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-17T07:11:11.787 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:11.787 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:12.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-17T07:11:12.621 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:00 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:12.766 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:12.767 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2023-10-17T07:11:12.767 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:12.767 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2023-10-17T07:11:12.768 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:12.768 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 7.4 M 2023-10-17T07:11:12.768 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:11:12.768 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-17T07:11:12.768 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-17T07:11:12.768 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-17T07:11:12.769 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-17T07:11:12.769 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-17T07:11:12.769 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-17T07:11:12.769 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-17T07:11:12.769 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-17T07:11:12.770 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-17T07:11:12.770 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-17T07:11:12.770 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-17T07:11:12.770 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:12.770 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:12.771 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2023-10-17T07:11:12.771 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2023-10-17T07:11:12.771 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:12.772 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 37 M 2023-10-17T07:11:12.772 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 209 M 2023-10-17T07:11:12.772 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 201 kB/s | 32 kB 00:00 2023-10-17T07:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2023-10-17T07:11:13.003 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 822 kB/s | 34 kB 00:00 2023-10-17T07:11:13.128 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2023-10-17T07:11:13.220 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 205 kB 00:00 2023-10-17T07:11:13.304 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-10-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2023-10-17T07:11:13.412 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-10-17T07:11:13.771 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2023-10-17T07:11:13.813 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2023-10-17T07:11:13.838 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2023-10-17T07:11:14.505 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 4.3 MB/s | 7.4 MB 00:01 2023-10-17T07:11:14.508 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:14.508 INFO:teuthology.orchestra.run.smithi049.stdout:Total 21 MB/s | 37 MB 00:01 2023-10-17T07:11:14.509 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:14.533 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:14.533 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:14.983 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:14.984 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:15.834 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:15.920 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-17T07:11:15.968 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-17T07:11:16.065 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-17T07:11:16.104 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-17T07:11:16.287 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-17T07:11:16.320 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-17T07:11:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-17T07:11:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-17T07:11:17.227 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-17T07:11:18.547 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-17T07:11:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-17T07:11:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6790.ga83d4 12/12 2023-10-17T07:11:22.284 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6790.ga83d4 12/12 2023-10-17T07:11:22.284 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6790.ga83d4 1/12 2023-10-17T07:11:22.284 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-17T07:11:22.284 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-17T07:11:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-17T07:11:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-17T07:11:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-17T07:11:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-17T07:11:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-17T07:11:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-17T07:11:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-17T07:11:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-17T07:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-17T07:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-17T07:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-17T07:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-17T07:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-17T07:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-17T07:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-17T07:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-17T07:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-17T07:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-17T07:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-17T07:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-17T07:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:23.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2023-10-17T07:11:23.645 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:11 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:23.792 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:23.793 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-10-17T07:11:23.793 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:23.793 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-10-17T07:11:23.793 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:23.794 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 47 k 2023-10-17T07:11:23.794 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:11:23.794 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-17T07:11:23.794 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-17T07:11:23.794 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-17T07:11:23.795 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-17T07:11:23.795 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-17T07:11:23.795 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-17T07:11:23.795 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-17T07:11:23.795 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-17T07:11:23.796 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-17T07:11:23.796 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-17T07:11:23.796 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-17T07:11:23.796 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-17T07:11:23.796 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:23.797 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:23.797 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-10-17T07:11:23.797 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2023-10-17T07:11:23.797 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:23.798 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.0 M 2023-10-17T07:11:23.798 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2023-10-17T07:11:23.798 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:24.236 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 136 kB/s | 24 kB 00:00 2023-10-17T07:11:24.261 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 18 kB 00:00 2023-10-17T07:11:24.395 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2023-10-17T07:11:24.420 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 833 kB/s | 132 kB 00:00 2023-10-17T07:11:24.496 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 211 kB 00:00 2023-10-17T07:11:24.521 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 43 kB 00:00 2023-10-17T07:11:24.780 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2023-10-17T07:11:24.805 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): ceph-mgr-rook-18.0.0-6790.ga83d4841.el9 64 kB/s | 47 kB 00:00 2023-10-17T07:11:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 96 kB/s | 32 kB 00:00 2023-10-17T07:11:24.997 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-google-auth-2.23.2-1.el9.noarc 905 kB/s | 196 kB 00:00 2023-10-17T07:11:25.081 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2023-10-17T07:11:25.122 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 202 kB/s | 59 kB 00:00 2023-10-17T07:11:25.156 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2023-10-17T07:11:25.160 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:25.160 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-10-17T07:11:25.161 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:25.175 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:25.175 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:25.288 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:25.289 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:25.822 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-17T07:11:26.067 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-17T07:11:26.123 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-17T07:11:26.159 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-17T07:11:26.201 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-17T07:11:26.296 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-17T07:11:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 7/13 2023-10-17T07:11:26.477 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-17T07:11:27.238 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-17T07:11:27.290 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-17T07:11:27.324 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-17T07:11:27.370 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-17T07:11:27.394 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:18.0.0-6790.ga83d4841.el9.noarch 13/13 2023-10-17T07:11:28.169 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6790.ga83d4841.el9.noarch 13/13 2023-10-17T07:11:28.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6790.ga83d4841.el9.noarch 1/13 2023-10-17T07:11:28.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-17T07:11:28.169 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-17T07:11:28.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-17T07:11:28.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-17T07:11:28.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-17T07:11:28.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-17T07:11:28.170 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-17T07:11:28.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-17T07:11:28.171 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 10/13 2023-10-17T07:11:28.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-17T07:11:28.172 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-17T07:11:28.537 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-17T07:11:28.537 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:28.537 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:28.538 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:28.538 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-17T07:11:28.538 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-17T07:11:28.538 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-17T07:11:28.538 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-17T07:11:28.539 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-17T07:11:28.539 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-17T07:11:28.539 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-17T07:11:28.539 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-17T07:11:28.539 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-17T07:11:28.540 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-17T07:11:28.540 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-17T07:11:28.540 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-17T07:11:28.540 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:28.540 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:28.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2023-10-17T07:11:29.388 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:17 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:29.533 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:29.534 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-10-17T07:11:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:29.534 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-10-17T07:11:29.534 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 132 k 2023-10-17T07:11:29.535 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:11:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-17T07:11:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-17T07:11:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-17T07:11:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-17T07:11:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-17T07:11:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:29.536 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:29.536 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2023-10-17T07:11:29.536 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-10-17T07:11:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:29.537 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 6.8 M 2023-10-17T07:11:29.537 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 33 M 2023-10-17T07:11:29.538 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:30.025 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 915 kB/s | 228 kB 00:00 2023-10-17T07:11:30.267 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2023-10-17T07:11:30.292 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6790.ga83d4841.e 256 kB/s | 132 kB 00:00 2023-10-17T07:11:30.393 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2023-10-17T07:11:30.418 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2023-10-17T07:11:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2023-10-17T07:11:30.445 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:30.445 INFO:teuthology.orchestra.run.smithi049.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2023-10-17T07:11:30.445 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:30.455 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:30.455 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:30.550 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:30.551 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-17T07:11:31.079 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-17T07:11:31.630 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-17T07:11:31.717 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-17T07:11:31.784 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-17T07:11:31.807 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 6/6 2023-10-17T07:11:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 6/6 2023-10-17T07:11:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 1/6 2023-10-17T07:11:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-17T07:11:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-17T07:11:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-17T07:11:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-17T07:11:32.903 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-17T07:11:32.903 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:32.904 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:32.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-17T07:11:32.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-17T07:11:32.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-17T07:11:32.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-17T07:11:32.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-17T07:11:32.905 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:33.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2023-10-17T07:11:33.763 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:21 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:33.909 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:33.910 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:33.910 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:33.910 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:33.910 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:33.910 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 832 k 2023-10-17T07:11:33.911 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T07:11:33.911 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-17T07:11:33.911 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:33.911 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:33.911 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:33.913 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2023-10-17T07:11:33.913 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 910 k 2023-10-17T07:11:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 3.0 M 2023-10-17T07:11:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:34.028 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 674 kB/s | 79 kB 00:00 2023-10-17T07:11:34.353 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-fuse-18.0.0-6790.ga83d4841.el9.x86_ 1.8 MB/s | 832 kB 00:00 2023-10-17T07:11:34.354 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:34.354 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.0 MB/s | 910 kB 00:00 2023-10-17T07:11:34.355 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:34.376 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:34.376 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:34.456 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:34.456 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:34.641 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:34.789 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-17T07:11:34.864 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 2/2 2023-10-17T07:11:35.574 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 2/2 2023-10-17T07:11:35.574 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 1/2 2023-10-17T07:11:35.883 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-17T07:11:35.883 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:35.883 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-17T07:11:35.884 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:36.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2023-10-17T07:11:36.768 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:24 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:36.915 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:36.916 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:36.916 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T07:11:36.916 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:36.917 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:18.0.0-6790.ga83d4841.el9 ceph-noarch 262 k 2023-10-17T07:11:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:36.917 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:36.917 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:36.918 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:36.918 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:36.918 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 262 k 2023-10-17T07:11:36.918 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.2 M 2023-10-17T07:11:36.918 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:37.135 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-volume-18.0.0-6790.ga83d4841.el9.noarch.rp 1.2 MB/s | 262 kB 00:00 2023-10-17T07:11:37.136 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:37.136 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.2 MB/s | 262 kB 00:00 2023-10-17T07:11:37.136 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:37.145 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:37.145 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:37.184 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:37.185 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:37.419 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:37.479 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:37.495 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:37.496 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:11:37.496 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-17T07:11:37.496 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:18.0.0-6790.ga83d4841.el9.noarch 1/1 2023-10-17T07:11:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:38.550 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:18.0.0-6790.ga83d4841.el9.noarch 2023-10-17T07:11:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:38.551 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:38.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2023-10-17T07:11:39.426 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:27 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:39.570 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:39.571 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T07:11:39.571 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:39.571 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 127 k 2023-10-17T07:11:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:39.572 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:39.572 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:39.572 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:39.573 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 127 k 2023-10-17T07:11:39.573 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 460 k 2023-10-17T07:11:39.573 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:39.891 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-18.0.0-6790.ga83d4841.el9.x86_64 398 kB/s | 127 kB 00:00 2023-10-17T07:11:39.891 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:39.892 INFO:teuthology.orchestra.run.smithi049.stdout:Total 396 kB/s | 127 kB 00:00 2023-10-17T07:11:39.892 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:39.898 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:39.898 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:39.956 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:39.956 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:40.114 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:40.224 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:40.695 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:41.001 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:41.001 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:41.002 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:41.002 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:11:41.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:41.002 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:41.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2023-10-17T07:11:41.894 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:29 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:41.987 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:42.039 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:42.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2023-10-17T07:11:42.717 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:30 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:42.863 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:42.863 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:42.863 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T07:11:42.864 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:42.864 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:42.864 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 31 k 2023-10-17T07:11:42.864 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:42.864 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:42.864 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:42.865 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:42.865 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:42.865 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 31 k 2023-10-17T07:11:42.865 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 139 k 2023-10-17T07:11:42.866 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-18.0.0-6790.ga83d4841.el9.x86_6 126 kB/s | 31 kB 00:00 2023-10-17T07:11:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:Total 126 kB/s | 31 kB 00:00 2023-10-17T07:11:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:43.115 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:43.115 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:43.143 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:43.144 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:43.352 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:43.761 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:44.021 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:11:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:44.022 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:44.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2023-10-17T07:11:44.806 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:32 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:44.898 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:44.950 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:44.952 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:44.952 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:45.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2023-10-17T07:11:45.629 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:33 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:45.721 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:45.775 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:45.776 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:45.776 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:45.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2023-10-17T07:11:46.442 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:34 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:46.534 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:46.587 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:46.588 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:46.588 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:46.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2023-10-17T07:11:47.264 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:35 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:47.355 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:47.408 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:47.409 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:47.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2023-10-17T07:11:48.085 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:36 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:48.178 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:48.231 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:48.231 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:48.232 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:48.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2023-10-17T07:11:48.914 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:36 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:49.007 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:18.0.0-6790.ga83d4841.el9.x86_64 is already installed. 2023-10-17T07:11:49.060 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:49.061 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T07:11:49.061 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:49.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2023-10-17T07:11:49.752 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:37 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:49.900 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:49.901 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:49.901 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-10-17T07:11:49.901 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:49.901 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:49.901 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 87 k 2023-10-17T07:11:49.902 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:49.902 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:49.902 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:49.902 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:49.902 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:49.903 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 87 k 2023-10-17T07:11:49.903 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 252 k 2023-10-17T07:11:49.903 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:50.097 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-18.0.0-6790.ga83d4841.el9.x86_64.rpm 444 kB/s | 87 kB 00:00 2023-10-17T07:11:50.098 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:50.098 INFO:teuthology.orchestra.run.smithi049.stdout:Total 440 kB/s | 87 kB 00:00 2023-10-17T07:11:50.099 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:50.106 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:50.106 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:50.164 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:50.164 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:50.306 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:50.406 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:51.150 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:51.150 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:51.150 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:51.151 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:11:51.151 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:51.151 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:51.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2023-10-17T07:11:52.014 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:40 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:52.159 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:52.160 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:52.160 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T07:11:52.161 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:52.161 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:52.161 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 3.0 M 2023-10-17T07:11:52.161 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:52.161 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:52.162 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:52.162 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:52.162 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:52.162 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.0 M 2023-10-17T07:11:52.162 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 13 M 2023-10-17T07:11:52.163 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:53.292 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-18.0.0-6790.ga83d4841.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2023-10-17T07:11:53.293 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:53.293 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.7 MB/s | 3.0 MB 00:01 2023-10-17T07:11:53.293 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:53.304 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:53.304 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:53.382 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:53.382 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:53.837 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:53.884 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:53.911 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:53.911 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-17T07:11:53.911 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-17T07:11:53.911 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-17T07:11:53.911 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-17T07:11:53.912 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:55.092 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:55.093 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:55.093 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:55.093 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:11:55.093 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:55.094 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:55.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2023-10-17T07:11:55.982 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:43 ago on Tue 17 Oct 2023 07:09:12 AM UTC. 2023-10-17T07:11:56.127 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T07:11:56.128 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:56.128 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-10-17T07:11:56.128 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:56.129 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T07:11:56.129 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:18.0.0-6790.ga83d4841.el9 ceph 169 k 2023-10-17T07:11:56.129 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:56.129 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T07:11:56.129 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T07:11:56.129 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T07:11:56.130 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:56.130 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 169 k 2023-10-17T07:11:56.130 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 533 k 2023-10-17T07:11:56.130 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T07:11:56.745 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-18.0.0-6790.ga83d4841.el9.x86_64.rpm 274 kB/s | 169 kB 00:00 2023-10-17T07:11:56.746 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T07:11:56.747 INFO:teuthology.orchestra.run.smithi049.stdout:Total 273 kB/s | 169 kB 00:00 2023-10-17T07:11:56.747 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T07:11:56.755 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T07:11:56.755 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T07:11:56.814 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T07:11:56.815 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T07:11:56.976 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T07:11:57.088 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:18.0.0-6790.ga83d4841.el9.x86_64 1/1 2023-10-17T07:11:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:57.856 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T07:11:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:18.0.0-6790.ga83d4841.el9.x86_64 2023-10-17T07:11:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:11:57.856 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T07:11:58.140 DEBUG:teuthology.parallel:result is None 2023-10-17T07:11:58.141 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a83d484198747052d1ed9eeed0a72e20b007e411 2023-10-17T07:11:58.373 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-17T07:11:58.394 INFO:teuthology.orchestra.run.smithi049.stdout:18.0.0-6790.ga83d4841.el9 2023-10-17T07:11:58.395 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6790.ga83d4841.el9 2023-10-17T07:11:58.395 INFO:teuthology.task.install:The correct ceph version 18.0.0-6790.ga83d4841 is installed. 2023-10-17T07:11:58.396 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-17T07:11:58.396 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:11:58.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-17T07:11:58.465 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-17T07:11:58.466 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:11:58.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2023-10-17T07:11:58.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-17T07:11:58.605 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-17T07:11:58.606 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:11:58.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-17T07:11:58.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-17T07:11:58.747 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-17T07:11:58.748 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:11:58.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2023-10-17T07:11:58.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-17T07:11:58.888 INFO:teuthology.run_tasks:Running task ceph... 2023-10-17T07:11:58.957 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-17T07:11:58.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2023-10-17T07:11:58.989 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-17T07:11:58.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-17T07:11:59.060 INFO:tasks.ceph:Creating extra log directories... 2023-10-17T07:11:59.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-17T07:11:59.138 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-17T07:11:59.138 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-17T07:11:59.139 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067', 'branch': 'wip-crimson-scrub-testing-2023-10-16', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7431067', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-crimson-scrub-testing-2023-10-16'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411'}}, 'workunit': {'branch': 'wip-crimson-scrub-testing-2023-10-16', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411'}}, 'owner': 'scheduled_sjust@teuthology', 'priority': 51, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 3210, 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-crimson-scrub-testing-2023-10-16', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkX6Uym4JX933EQyRfA93ZVWZvAPT32MnwBP+A/EbyOtGI3nWlbNM/lVWzrn3EPzD/zsLqup9dsaX5G55qcH1Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': 'a83d484198747052d1ed9eeed0a72e20b007e411', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}, 'env': {'RBD_FEATURES': '61'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-17_06:50:25', 'tube': 'smithi', 'user': 'sjust', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-17T07:11:59.139 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-17T07:11:59.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2023-10-17T07:11:59.256 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:11:59.257 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2023-10-17T07:11:59.312 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-17T07:11:59.312 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2023-10-17T07:11:59.367 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-17T07:11:59.367 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T07:11:59.368 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-10-17T07:11:59.368 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T07:11:59.368 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T07:11:59.368 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 07:11:54.168602934 +0000 2023-10-17T07:11:59.368 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 07:08:21.912141487 +0000 2023-10-17T07:11:59.369 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 07:08:21.912141487 +0000 2023-10-17T07:11:59.369 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-10-17 07:08:21.912141487 +0000 2023-10-17T07:11:59.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-17T07:11:59.436 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T07:11:59.437 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T07:11:59.437 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 7.738e-05 s, 6.6 MB/s 2023-10-17T07:11:59.438 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-17T07:11:59.494 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2023-10-17T07:11:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-17T07:11:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T07:11:59.550 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-10-17T07:11:59.550 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T07:11:59.551 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T07:11:59.551 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 07:11:54.168602934 +0000 2023-10-17T07:11:59.551 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 07:08:22.411127742 +0000 2023-10-17T07:11:59.551 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 07:08:22.411127742 +0000 2023-10-17T07:11:59.552 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-10-17 07:08:22.411127742 +0000 2023-10-17T07:11:59.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-17T07:11:59.619 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T07:11:59.620 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T07:11:59.620 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.4875e-05 s, 5.4 MB/s 2023-10-17T07:11:59.621 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-17T07:11:59.677 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2023-10-17T07:11:59.732 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-17T07:11:59.732 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T07:11:59.732 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-17T07:11:59.733 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T07:11:59.733 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T07:11:59.733 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 07:11:54.168602934 +0000 2023-10-17T07:11:59.733 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 07:08:22.913113915 +0000 2023-10-17T07:11:59.734 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 07:08:22.913113915 +0000 2023-10-17T07:11:59.734 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-10-17 07:08:22.913113915 +0000 2023-10-17T07:11:59.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-17T07:11:59.801 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T07:11:59.801 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T07:11:59.802 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 8.9222e-05 s, 5.7 MB/s 2023-10-17T07:11:59.802 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-17T07:11:59.857 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2023-10-17T07:11:59.912 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-17T07:11:59.912 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T07:11:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-17T07:11:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T07:11:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T07:11:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 07:11:54.169602907 +0000 2023-10-17T07:11:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 07:08:23.407100308 +0000 2023-10-17T07:11:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 07:08:23.407100308 +0000 2023-10-17T07:11:59.914 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2023-10-17 07:08:23.407100308 +0000 2023-10-17T07:11:59.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-17T07:11:59.981 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T07:11:59.981 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T07:11:59.981 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.6349e-05 s, 5.3 MB/s 2023-10-17T07:11:59.984 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-17T07:12:00.040 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T07:12:00.040 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T07:12:00.041 INFO:tasks.ceph:Generating config... 2023-10-17T07:12:00.042 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-17T07:12:00.042 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-17T07:12:00.043 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-17T07:12:00.043 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-17T07:12:00.043 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-17T07:12:00.043 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-17T07:12:00.044 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-17T07:12:00.044 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-17T07:12:00.044 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-17T07:12:00.045 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-17T07:12:00.045 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-17T07:12:00.045 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-17T07:12:00.046 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-17T07:12:00.046 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-17T07:12:00.046 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-17T07:12:00.046 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-17T07:12:00.047 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-17T07:12:00.047 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-17T07:12:00.047 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-17T07:12:00.048 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-17T07:12:00.048 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-17T07:12:00.048 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-17T07:12:00.049 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-17T07:12:00.049 INFO:tasks.ceph:Setting up mon.a... 2023-10-17T07:12:00.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-17T07:12:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.keyring 2023-10-17T07:12:00.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-17T07:12:00.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-17T07:12:00.227 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.49')] 2023-10-17T07:12:00.227 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.49', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-17T07:12:00.228 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:00.229 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T07:12:00.282 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.49 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:00.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:00.367+0000 7f1a8422fe80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:00.367+0000 7f1a8422fe80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:00.367+0000 7f1a8422fe80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:00.369 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:00.369 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid f2b3b845-1504-4f19-98a2-7647538e104c 2023-10-17T07:12:00.370 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = pacific 2023-10-17T07:12:00.370 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2023-10-17T07:12:00.370 INFO:teuthology.orchestra.run.smithi049.stdout:fsid f2b3b845-1504-4f19-98a2-7647538e104c 2023-10-17T07:12:00.370 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2023-10-17T07:12:00.368008+0000 2023-10-17T07:12:00.371 INFO:teuthology.orchestra.run.smithi049.stdout:created 2023-10-17T07:12:00.368008+0000 2023-10-17T07:12:00.371 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 16 (pacific) 2023-10-17T07:12:00.371 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 1 2023-10-17T07:12:00.371 INFO:teuthology.orchestra.run.smithi049.stdout:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2023-10-17T07:12:00.371 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-17T07:12:00.372 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T07:12:00.388 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f2b3b845-1504-4f19-98a2-7647538e104c... 2023-10-17T07:12:00.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-17T07:12:00.488 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2023-10-17T07:12:00.488 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2023-10-17T07:12:00.489 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-17T07:12:00.489 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = cephx 2023-10-17T07:12:00.489 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.489 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2023-10-17T07:12:00.489 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.490 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2023-10-17T07:12:00.490 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.490 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2023-10-17T07:12:00.490 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2023-10-17T07:12:00.490 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.491 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2023-10-17T07:12:00.491 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2023-10-17T07:12:00.491 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.491 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-17T07:12:00.492 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2023-10-17T07:12:00.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.492 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2023-10-17T07:12:00.492 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-17T07:12:00.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.492 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2023-10-17T07:12:00.493 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.493 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2023-10-17T07:12:00.493 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2023-10-17T07:12:00.493 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2023-10-17T07:12:00.494 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2023-10-17T07:12:00.494 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2023-10-17T07:12:00.494 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2023-10-17T07:12:00.494 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2023-10-17T07:12:00.494 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-17T07:12:00.495 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-17T07:12:00.495 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2023-10-17T07:12:00.495 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.495 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-17T07:12:00.495 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.495 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2023-10-17T07:12:00.496 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.496 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2023-10-17T07:12:00.496 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.496 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2023-10-17T07:12:00.497 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2023-10-17T07:12:00.497 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2023-10-17T07:12:00.497 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = 172.21.15.49 2023-10-17T07:12:00.497 INFO:teuthology.orchestra.run.smithi049.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-17T07:12:00.497 INFO:teuthology.orchestra.run.smithi049.stdout: ms client mode = crc 2023-10-17T07:12:00.498 INFO:teuthology.orchestra.run.smithi049.stdout: ms cluster mode = crc 2023-10-17T07:12:00.498 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon client mode = crc 2023-10-17T07:12:00.498 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon cluster mode = crc 2023-10-17T07:12:00.498 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon service mode = crc 2023-10-17T07:12:00.498 INFO:teuthology.orchestra.run.smithi049.stdout: ms service mode = crc 2023-10-17T07:12:00.499 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = f2b3b845-1504-4f19-98a2-7647538e104c 2023-10-17T07:12:00.499 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.499 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2023-10-17T07:12:00.499 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2023-10-17T07:12:00.500 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.500 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2023-10-17T07:12:00.500 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2023-10-17T07:12:00.501 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2023-10-17T07:12:00.501 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.501 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2023-10-17T07:12:00.501 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2023-10-17T07:12:00.501 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.501 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2023-10-17T07:12:00.502 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2023-10-17T07:12:00.502 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2023-10-17T07:12:00.502 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.502 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2023-10-17T07:12:00.503 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2023-10-17T07:12:00.503 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.503 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2023-10-17T07:12:00.503 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.503 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2023-10-17T07:12:00.503 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.504 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2023-10-17T07:12:00.504 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.504 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2023-10-17T07:12:00.504 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2023-10-17T07:12:00.504 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.505 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2023-10-17T07:12:00.505 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2023-10-17T07:12:00.505 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 20 2023-10-17T07:12:00.505 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar smp = 3 2023-10-17T07:12:00.505 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T07:12:00.505 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 20 2023-10-17T07:12:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: osd shutdown pgref assert = True 2023-10-17T07:12:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2023-10-17T07:12:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T07:12:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2023-10-17T07:12:00.507 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-17T07:12:00.507 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-17T07:12:00.507 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-17T07:12:00.507 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-17T07:12:00.507 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2023-10-17T07:12:00.508 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.508 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2023-10-17T07:12:00.508 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T07:12:00.508 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-17T07:12:00.508 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 20 2023-10-17T07:12:00.508 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2023-10-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2023-10-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2023-10-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2023-10-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-17T07:12:00.510 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.510 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-17T07:12:00.510 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2023-10-17T07:12:00.510 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2023-10-17T07:12:00.510 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.511 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2023-10-17T07:12:00.511 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-17T07:12:00.511 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-17T07:12:00.511 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.511 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2023-10-17T07:12:00.512 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2023-10-17T07:12:00.512 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.512 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2023-10-17T07:12:00.512 INFO:teuthology.orchestra.run.smithi049.stdout: mon min osdmap epochs = 50 2023-10-17T07:12:00.512 INFO:teuthology.orchestra.run.smithi049.stdout: mon osdmap full prune interval = 2 2023-10-17T07:12:00.513 INFO:teuthology.orchestra.run.smithi049.stdout: mon osdmap full prune min = 15 2023-10-17T07:12:00.513 INFO:teuthology.orchestra.run.smithi049.stdout: mon osdmap full prune txsize = 2 2023-10-17T07:12:00.513 INFO:teuthology.orchestra.run.smithi049.stdout: paxos service trim min = 10 2023-10-17T07:12:00.513 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default crimson = True 2023-10-17T07:12:00.513 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:00.513 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2023-10-17T07:12:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2023-10-17T07:12:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2023-10-17T07:12:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2023-10-17T07:12:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-17T07:12:00.514 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-17T07:12:00.515 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2023-10-17T07:12:00.516 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-17T07:12:00.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-17T07:12:00.582 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-17T07:12:00.583 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:00.583 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-17T07:12:00.598 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:00.599 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-17T07:12:00.653 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T07:12:00.654 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:00.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-17T07:12:00.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-17T07:12:00.741 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:00.741 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:00.796 INFO:tasks.ceph:Setting up mon nodes... 2023-10-17T07:12:00.796 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-17T07:12:00.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-17T07:12:00.903 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-17T07:12:00.906 INFO:tasks.ceph:Setting up mds nodes... 2023-10-17T07:12:00.907 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-17T07:12:00.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-17T07:12:00.959 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-17T07:12:00.977 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-17T07:12:00.978 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T07:12:00.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-17T07:12:01.009 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T07:12:01.009 INFO:tasks.ceph:role: osd.0 2023-10-17T07:12:01.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T07:12:01.010 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-17T07:12:01.079 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T07:12:01.079 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T07:12:01.079 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T07:12:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-17T07:12:01.080 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T07:12:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-17T07:12:01.080 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T07:12:01.080 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-17T07:12:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T07:12:01.081 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T07:12:01.116 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-17T07:12:01.118 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-17T07:12:01.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-17T07:12:01.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-17T07:12:01.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-17T07:12:01.341 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T07:12:01.342 INFO:tasks.ceph:role: osd.1 2023-10-17T07:12:01.342 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T07:12:01.342 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-17T07:12:01.412 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T07:12:01.413 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T07:12:01.413 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T07:12:01.413 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-17T07:12:01.413 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T07:12:01.413 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-17T07:12:01.413 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T07:12:01.414 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-17T07:12:01.414 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T07:12:01.414 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T07:12:01.447 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-17T07:12:01.449 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-17T07:12:01.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-17T07:12:01.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-17T07:12:01.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-17T07:12:01.671 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T07:12:01.672 INFO:tasks.ceph:role: osd.2 2023-10-17T07:12:01.672 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T07:12:01.672 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-17T07:12:01.741 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T07:12:01.742 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T07:12:01.742 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T07:12:01.742 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-17T07:12:01.742 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T07:12:01.742 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-17T07:12:01.743 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T07:12:01.743 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-17T07:12:01.743 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T07:12:01.743 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T07:12:01.774 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-17T07:12:01.776 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-17T07:12:01.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-17T07:12:01.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-17T07:12:01.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:02.499 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-17T07:12:02.500 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,500 seastar - Reactor backend: linux-aio 2023-10-17T07:12:02.517 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,517 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-17T07:12:02.517 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,517 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-17T07:12:02.517 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,517 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-17T07:12:02.518 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,517 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-17T07:12:02.519 INFO:teuthology.orchestra.run.smithi049.stderr:==80223==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-17T07:12:02.525 INFO:teuthology.orchestra.run.smithi049.stderr:==80223==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffddace000; bottom 0x7f0d16428000; size: 0x00f2c76a6000 (1042727723008) 2023-10-17T07:12:02.525 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-10-17T07:12:02.526 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-17T07:12:02.561 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,561 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-17T07:12:02.562 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:02,561 [shard 0] osd - get_early_config: set --smp 3 2023-10-17T07:12:03.076 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:03.077 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-10-17T07:12:03.077 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-10-17T07:12:03.077 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-10-17T07:12:03.077 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:03.078 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T07:12:03.131 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-17T07:12:03.132 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:03,132 seastar - Reactor backend: linux-aio 2023-10-17T07:12:03.150 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:03,149 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-17T07:12:03.150 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:03,149 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-17T07:12:03.150 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:03,149 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-17T07:12:03.157 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:03,157 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-17T07:12:03.160 INFO:teuthology.orchestra.run.smithi049.stderr:==80221==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-17T07:12:03.166 INFO:teuthology.orchestra.run.smithi049.stderr:==80221==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffddace000; bottom 0x7f0d14211000; size: 0x00f2c98bd000 (1042763468800) 2023-10-17T07:12:03.166 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-10-17T07:12:03.166 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-17T07:12:03.248 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d5d 0x2d35bd66 0x2d35d114 0x2d35e662 0x2d35e9e6 0x2d35eb30 0x2d35ef86 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26f70ec4 0x26f7350d 0x26f7531e 0x26f77b83 0x26d8a5b9 0x1f56e37e 0x1f4e3049 0x1f4e6fb5 0x2d6d2929 2023-10-17T07:12:03.248 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2023-10-17T07:12:06.783 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:06.783 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-10-17T07:12:06.784 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-10-17T07:12:06.784 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-10-17T07:12:06.784 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:06.784 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T07:12:06.785 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid f2b3b845-1504-4f19-98a2-7647538e104c 2023-10-17T07:12:06.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-17T07:12:06.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:07.404 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-17T07:12:07.405 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,405 seastar - Reactor backend: linux-aio 2023-10-17T07:12:07.422 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,422 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-17T07:12:07.422 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,422 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-17T07:12:07.422 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,422 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-17T07:12:07.422 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,422 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-17T07:12:07.424 INFO:teuthology.orchestra.run.smithi049.stderr:==81032==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-17T07:12:07.430 INFO:teuthology.orchestra.run.smithi049.stderr:==81032==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe344ea000; bottom 0x7f2d8b96b000; size: 0x00d0a8b7f000 (896183824384) 2023-10-17T07:12:07.431 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-10-17T07:12:07.431 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-17T07:12:07.456 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d35bd66 0x2d35d114 0x2d35e662 0x2d35e9e6 0x2d35eb30 0x2d35ef86 0x54daf 0x23785b89 0x237d8be0 0x237feee9 0x236a855e 0x236d5fb1 0x256a4021 0x256a9749 0x256abce4 0x256acebd 0x23265577 0x232b4336 0x232d9bd9 0x2d300e4b 0x2d39620e 0x2d5b3184 0x2d5b4fcf 0x2d02a330 0x2d02c231 0x1f62fdd8 0x1f639f45 0x1f4b31db 0x3feaf 0x3ff5f 0x1ef6ca04 2023-10-17T07:12:07.457 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2023-10-17T07:12:07.466 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,466 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-17T07:12:07.467 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:07,466 [shard 0] osd - get_early_config: set --smp 3 2023-10-17T07:12:07.976 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:07.977 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-10-17T07:12:07.977 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-10-17T07:12:07.977 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-10-17T07:12:07.977 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:07.978 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T07:12:08.031 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-17T07:12:08.032 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:08,032 seastar - Reactor backend: linux-aio 2023-10-17T07:12:08.049 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:08,049 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-17T07:12:08.050 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:08,049 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-17T07:12:08.050 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:08,049 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-17T07:12:08.057 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:08,057 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-17T07:12:08.060 INFO:teuthology.orchestra.run.smithi049.stderr:==81030==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-17T07:12:08.066 INFO:teuthology.orchestra.run.smithi049.stderr:==81030==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe344ea000; bottom 0x7f2d89754000; size: 0x00d0aad96000 (896219570176) 2023-10-17T07:12:08.066 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-10-17T07:12:08.066 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-17T07:12:08.148 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d5d 0x2d35bd66 0x2d35d114 0x2d35e662 0x2d35e9e6 0x2d35eb30 0x2d35ef86 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26f70ec4 0x26f7350d 0x26f7531e 0x26f77b83 0x26d8a5b9 0x1f56e37e 0x1f4e3049 0x1f4e6fb5 0x2d6d2929 2023-10-17T07:12:08.148 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8bb4fa43 0xffffffff8bb4fa8a 0xffffffff8c65918c 0xffffffff8c8000aa 2023-10-17T07:12:11.691 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:11.691 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-10-17T07:12:11.691 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-10-17T07:12:11.692 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-10-17T07:12:11.692 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:11.692 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T07:12:11.692 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid f2b3b845-1504-4f19-98a2-7647538e104c 2023-10-17T07:12:11.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-17T07:12:11.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:12.325 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-17T07:12:12.326 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,326 seastar - Reactor backend: linux-aio 2023-10-17T07:12:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,343 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-17T07:12:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,343 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-17T07:12:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,343 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-17T07:12:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,343 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-17T07:12:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:==81840==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-17T07:12:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:==81840==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc208c0000; bottom 0x7fdee3bbf000; size: 0x001d3cd01000 (125574320128) 2023-10-17T07:12:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-10-17T07:12:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-17T07:12:12.377 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d35bd66 0x2d35d114 0x2d35e662 0x2d35e9e6 0x2d35eb30 0x2d35ef86 0x54daf 0xad7f6 0x2387a8e3 0x2387bdda 0x23883009 0x236ac157 0x236d5530 0x256a4021 0x256a9749 0x256abce4 0x256acebd 0x23265577 0x232b4336 0x232d9bd9 0x2d300e4b 0x2d39620e 0x2d5b3184 0x2d5b4fcf 0x2d02a330 0x2d02c231 0x1f62fdd8 0x1f639f45 0x1f4b31db 0x3feaf 0x3ff5f 0x1ef6ca04 2023-10-17T07:12:12.377 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8bbab690 0xffffffff8bd73374 0xffffffff8ba89454 0xffffffff8c65d562 0xffffffff8c800b82 2023-10-17T07:12:12.387 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,387 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-17T07:12:12.387 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,387 [shard 0] osd - get_early_config: set --smp 3 2023-10-17T07:12:12.888 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:12.888 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-10-17T07:12:12.888 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-10-17T07:12:12.889 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-10-17T07:12:12.889 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:12.889 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T07:12:12.939 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-17T07:12:12.939 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,939 seastar - Reactor backend: linux-aio 2023-10-17T07:12:12.957 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,957 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-17T07:12:12.958 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,957 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-17T07:12:12.958 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,957 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-17T07:12:12.965 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2023-10-17 07:12:12,965 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-17T07:12:12.967 INFO:teuthology.orchestra.run.smithi049.stderr:==81838==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-17T07:12:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:==81838==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc208c0000; bottom 0x7fdee19a8000; size: 0x001d3ef18000 (125610065920) 2023-10-17T07:12:12.974 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2023-10-17T07:12:12.974 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-17T07:12:13.056 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d35bd66 0x2d35d114 0x2d35e662 0x2d35e9e6 0x2d35eb30 0x2d35ef86 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26f70ec4 0x26f7350d 0x26f7531e 0x26f77b83 0x26d8a5b9 0x1f56e37e 0x1f4e3049 0x1f4e6fb5 0x2d6d2929 2023-10-17T07:12:13.056 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8c670140 2023-10-17T07:12:16.601 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:16.602 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2023-10-17T07:12:16.602 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2023-10-17T07:12:16.602 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2023-10-17T07:12:16.602 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2023-10-17T07:12:16.602 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T07:12:16.603 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid f2b3b845-1504-4f19-98a2-7647538e104c 2023-10-17T07:12:16.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-17T07:12:16.672 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-17T07:12:16.672 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:16.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-17T07:12:16.743 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:16.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-17T07:12:16.813 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:16.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-17T07:12:16.883 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:16.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-17T07:12:16.953 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:16.953 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-17T07:12:17.008 INFO:tasks.ceph:Adding keys to all mons... 2023-10-17T07:12:17.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-17T07:12:17.077 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2023-10-17T07:12:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDAMy5lvqrhNRAA95tbMDQouvyZwmvYF/J5Eg== 2023-10-17T07:12:17.077 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2023-10-17T07:12:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDDMy5lbo3VDBAAbQTgltTWkT5We7cymgnC/w== 2023-10-17T07:12:17.078 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2023-10-17T07:12:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDIMy5lKd3fBhAAHnmxMImESsH5XqTNzvqTbw== 2023-10-17T07:12:17.078 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2023-10-17T07:12:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDNMy5l21hfARAAELA6roR5u9lQ1SbwzCRijA== 2023-10-17T07:12:17.078 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2023-10-17T07:12:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDAMy5lr40tORAAnF+UpFVeVFTtH6n3DpmSvA== 2023-10-17T07:12:17.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-17T07:12:17.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T07:12:17.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T07:12:17.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T07:12:17.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-17T07:12:17.432 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-17T07:12:17.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-17T07:12:17.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-17T07:12:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:17.566+0000 7f59dfa2e980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:17.566+0000 7f59dfa2e980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-17T07:12:17.614 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-17T07:12:17.669 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-17T07:12:17.669 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-17T07:12:17.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-17T07:12:17.713 INFO:tasks.ceph.mon.a:Started 2023-10-17T07:12:17.714 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-17T07:12:17.714 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-17T07:12:17.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-17T07:12:17.716 INFO:tasks.ceph.mgr.x:Started 2023-10-17T07:12:17.717 INFO:tasks.ceph:Setting crush tunables to default 2023-10-17T07:12:17.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd crush tunables default 2023-10-17T07:12:17.797 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:12:17.795+0000 7f5c96748980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.797 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:12:17.795+0000 7f5c96748980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.797 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:12:17.796+0000 7f5c96748980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.800 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:17.799+0000 7fd7f8c09f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:17.832+0000 7ff6eeb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:17.858+0000 7ff6eeb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:17.931 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2023-10-17T07:12:17.942 INFO:tasks.ceph:check_enable_crimson: True 2023-10-17T07:12:17.942 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-17T07:12:17.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-17T07:12:18.005 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:18.004+0000 7fd7f8c09f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:18.005 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:18.004+0000 7fd7f8c09f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:18.053 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:18.047+0000 7f63e319e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:18.055+0000 7f63e319e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:18.139 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:12:18.138+0000 7f5c8e9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-17T07:12:18.140 INFO:tasks.ceph.mon.a.smithi049.stderr:Please be aware that this feature is experimental, untested, 2023-10-17T07:12:18.140 INFO:tasks.ceph.mon.a.smithi049.stderr:unsupported, and may result in data corruption, data loss, 2023-10-17T07:12:18.140 INFO:tasks.ceph.mon.a.smithi049.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-17T07:12:18.140 INFO:tasks.ceph.mon.a.smithi049.stderr:feature with important data. 2023-10-17T07:12:18.140 INFO:tasks.ceph.mon.a.smithi049.stderr: 2023-10-17T07:12:18.465 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:18.464+0000 7fd7f8c09f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-17T07:12:18.518 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:18.517+0000 7fd7f8c09f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-17T07:12:18.944 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-17T07:12:18.945 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:18.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-17T07:12:18.976 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:18.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-17T07:12:19.047 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:12:19.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-17T07:12:19.075 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.074+0000 7fd7f8c09f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-17T07:12:19.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 998ba9c4-abb7-4433-82d2-b07a91a32553 0 2023-10-17T07:12:19.132 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.131+0000 7fd7f8c09f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-17T07:12:19.180 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.179+0000 7fd7f8c09f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-17T07:12:19.238 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.236+0000 7fd7f8c09f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-17T07:12:19.242 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:19.235+0000 7ffb3cb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:19.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:19.244+0000 7ffb3cb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:19.285 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.284+0000 7fd7f8c09f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-17T07:12:19.327 INFO:teuthology.orchestra.run.smithi049.stdout:0 2023-10-17T07:12:19.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 22b65ecf-81d9-4ed8-8289-b1ed12048278 1 2023-10-17T07:12:19.393 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.392+0000 7fd7f8c09f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-17T07:12:19.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:19.441+0000 7fa29fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:19.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:19.450+0000 7fa29fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:19.471 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.470+0000 7fd7f8c09f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-17T07:12:19.519 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.517+0000 7fd7f8c09f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-17T07:12:19.533 INFO:teuthology.orchestra.run.smithi049.stdout:1 2023-10-17T07:12:19.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 9e52f463-58e1-49f5-b361-a12594f3f054 2 2023-10-17T07:12:19.577 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.575+0000 7fd7f8c09f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-17T07:12:19.628 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:19.626+0000 7fd7f8c09f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-17T07:12:19.655 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:19.648+0000 7f1b1cd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:19.658 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:19.657+0000 7f1b1cd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:19.737 INFO:teuthology.orchestra.run.smithi049.stdout:2 2023-10-17T07:12:19.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-17T07:12:19.824 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-17T07:12:19.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-17T07:12:19.866 INFO:tasks.ceph.osd.0:Started 2023-10-17T07:12:19.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-17T07:12:19.902 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-17T07:12:19.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-17T07:12:19.944 INFO:tasks.ceph.osd.1:Started 2023-10-17T07:12:19.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-17T07:12:19.983 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-17T07:12:19.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-17T07:12:20.026 INFO:tasks.ceph.osd.2:Started 2023-10-17T07:12:20.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T07:12:20.056 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.054+0000 7fd7f8c09f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-17T07:12:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:20.175+0000 7f7d05f08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:20.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:20.184+0000 7f7d05f08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:20.229 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.227+0000 7fd7f8c09f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-17T07:12:20.258 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:20.259 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":6,"fsid":"f2b3b845-1504-4f19-98a2-7647538e104c","created":"2023-10-17T07:12:17.847825+0000","modified":"2023-10-17T07:12:19.736713+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T07:12:19.736713+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"998ba9c4-abb7-4433-82d2-b07a91a32553","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22b65ecf-81d9-4ed8-8289-b1ed12048278","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e52f463-58e1-49f5-b361-a12594f3f054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T07:12:20.274 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-17T07:12:20.275 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-17T07:12:20.370 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.368+0000 7fd7f8c09f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-17T07:12:20.425 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.424+0000 7fd7f8c09f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-17T07:12:20.539 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.537+0000 7fd7f8c09f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-17T07:12:20.577 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-17T07:12:20.647 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.645+0000 7fd7f8c09f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-17T07:12:20.679 INFO:teuthology.misc.health.smithi049.stderr:2023-10-17T07:12:20.674+0000 7f097fb55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:20.683 INFO:teuthology.misc.health.smithi049.stderr:2023-10-17T07:12:20.681+0000 7f097fb55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:20.753 INFO:teuthology.misc.health.smithi049.stdout: 2023-10-17T07:12:20.753 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":6,"fsid":"f2b3b845-1504-4f19-98a2-7647538e104c","created":"2023-10-17T07:12:17.847825+0000","modified":"2023-10-17T07:12:19.736713+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T07:12:19.736713+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"998ba9c4-abb7-4433-82d2-b07a91a32553","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22b65ecf-81d9-4ed8-8289-b1ed12048278","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e52f463-58e1-49f5-b361-a12594f3f054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T07:12:20.763 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-17T07:12:20.806 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.805+0000 7fd7f8c09f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-17T07:12:20.853 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:20.851+0000 7fd7f8c09f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-17T07:12:21.007 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:21.005+0000 7fd7f8c09f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-17T07:12:21.882 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:21.880+0000 7fd7f8c09f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-17T07:12:22.176 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:22.174+0000 7fd7f8c09f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-17T07:12:22.241 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:22.240+0000 7fd7f8c09f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-17T07:12:22.364 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:22.363+0000 7fd7f8c09f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-17T07:12:23.385 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:12:23.383+0000 7fd798e1a640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-17T07:12:27.066 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-17T07:12:27.178 INFO:teuthology.misc.health.smithi049.stderr:2023-10-17T07:12:27.173+0000 7f3db979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:27.181 INFO:teuthology.misc.health.smithi049.stderr:2023-10-17T07:12:27.180+0000 7f3db979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:27.371 INFO:teuthology.misc.health.smithi049.stdout: 2023-10-17T07:12:27.371 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":9,"fsid":"f2b3b845-1504-4f19-98a2-7647538e104c","created":"2023-10-17T07:12:17.847825+0000","modified":"2023-10-17T07:12:26.488530+0000","last_up_change":"2023-10-17T07:12:24.432340+0000","last_in_change":"2023-10-17T07:12:19.736713+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T07:12:26.438300+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"998ba9c4-abb7-4433-82d2-b07a91a32553","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":1249808404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":1249808404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":1249808404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":1249808404}]},"public_addr":"172.21.15.49:6802/1249808404","cluster_addr":"172.21.15.49:6803/1249808404","heartbeat_back_addr":"172.21.15.49:6809/1249808404","heartbeat_front_addr":"172.21.15.49:6808/1249808404","state":["exists","up"]},{"osd":1,"uuid":"22b65ecf-81d9-4ed8-8289-b1ed12048278","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":437182577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":437182577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":437182577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":437182577}]},"public_addr":"172.21.15.49:6805/437182577","cluster_addr":"172.21.15.49:6804/437182577","heartbeat_back_addr":"172.21.15.49:6810/437182577","heartbeat_front_addr":"172.21.15.49:6811/437182577","state":["exists","up"]},{"osd":2,"uuid":"9e52f463-58e1-49f5-b361-a12594f3f054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2615227823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":2615227823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2615227823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":2615227823}]},"public_addr":"172.21.15.49:6806/2615227823","cluster_addr":"172.21.15.49:6807/2615227823","heartbeat_back_addr":"172.21.15.49:6812/2615227823","heartbeat_front_addr":"172.21.15.49:6813/2615227823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T07:12:27.381 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-17T07:12:27.382 INFO:tasks.ceph:Creating RBD pool 2023-10-17T07:12:27.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-17T07:12:27.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:27.485+0000 7f473655e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:27.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:27.496+0000 7f473655e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:28.500 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2023-10-17T07:12:28.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-17T07:12:28.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:28.622+0000 7fc3a2b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:28.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:28.633+0000 7fc3a2b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:29.501 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-17T07:12:29.521 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-17T07:12:29.521 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-17T07:12:29.521 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-17T07:12:29.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-17T07:12:29.522 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-17T07:12:29.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:29.652+0000 7f74b6ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:29.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:29.659+0000 7f74b6ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:29.897 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:29.908 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":3249668619},{"type":"v1","addr":"172.21.15.49:6801","nonce":3249668619}]},"active_addr":"172.21.15.49:6801/3249668619","active_change":"2023-10-17T07:12:22.377346+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1304972469}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3300814055}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":365953043}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3614428099}]}]} 2023-10-17T07:12:29.911 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-17T07:12:29.911 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-17T07:12:29.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T07:12:30.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.018+0000 7fa53622d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.024+0000 7fa53622d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.214 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:30.215 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"fsid":"f2b3b845-1504-4f19-98a2-7647538e104c","created":"2023-10-17T07:12:17.847825+0000","modified":"2023-10-17T07:12:29.500043+0000","last_up_change":"2023-10-17T07:12:24.432340+0000","last_in_change":"2023-10-17T07:12:19.736713+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T07:12:26.438300+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T07:12:27.724280+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"998ba9c4-abb7-4433-82d2-b07a91a32553","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":1249808404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":1249808404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":1249808404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":1249808404}]},"public_addr":"172.21.15.49:6802/1249808404","cluster_addr":"172.21.15.49:6803/1249808404","heartbeat_back_addr":"172.21.15.49:6809/1249808404","heartbeat_front_addr":"172.21.15.49:6808/1249808404","state":["exists","up"]},{"osd":1,"uuid":"22b65ecf-81d9-4ed8-8289-b1ed12048278","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":437182577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":437182577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":437182577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":437182577}]},"public_addr":"172.21.15.49:6805/437182577","cluster_addr":"172.21.15.49:6804/437182577","heartbeat_back_addr":"172.21.15.49:6810/437182577","heartbeat_front_addr":"172.21.15.49:6811/437182577","state":["exists","up"]},{"osd":2,"uuid":"9e52f463-58e1-49f5-b361-a12594f3f054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2615227823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":2615227823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2615227823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":2615227823}]},"public_addr":"172.21.15.49:6806/2615227823","cluster_addr":"172.21.15.49:6807/2615227823","heartbeat_back_addr":"172.21.15.49:6812/2615227823","heartbeat_front_addr":"172.21.15.49:6813/2615227823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T07:12:30.227 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-17T07:12:30.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-17T07:12:30.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.333+0000 7ff34f82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.339+0000 7ff34f82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.533 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:30.533 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":13,"fsid":"f2b3b845-1504-4f19-98a2-7647538e104c","created":"2023-10-17T07:12:17.847825+0000","modified":"2023-10-17T07:12:30.505218+0000","last_up_change":"2023-10-17T07:12:24.432340+0000","last_in_change":"2023-10-17T07:12:19.736713+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T07:12:26.438300+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T07:12:27.724280+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"998ba9c4-abb7-4433-82d2-b07a91a32553","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":1249808404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":1249808404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":1249808404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":1249808404}]},"public_addr":"172.21.15.49:6802/1249808404","cluster_addr":"172.21.15.49:6803/1249808404","heartbeat_back_addr":"172.21.15.49:6809/1249808404","heartbeat_front_addr":"172.21.15.49:6808/1249808404","state":["exists","up"]},{"osd":1,"uuid":"22b65ecf-81d9-4ed8-8289-b1ed12048278","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":437182577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":437182577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":437182577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":437182577}]},"public_addr":"172.21.15.49:6805/437182577","cluster_addr":"172.21.15.49:6804/437182577","heartbeat_back_addr":"172.21.15.49:6810/437182577","heartbeat_front_addr":"172.21.15.49:6811/437182577","state":["exists","up"]},{"osd":2,"uuid":"9e52f463-58e1-49f5-b361-a12594f3f054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2615227823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":2615227823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2615227823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":2615227823}]},"public_addr":"172.21.15.49:6806/2615227823","cluster_addr":"172.21.15.49:6807/2615227823","heartbeat_back_addr":"172.21.15.49:6812/2615227823","heartbeat_front_addr":"172.21.15.49:6813/2615227823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T07:12:30.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-17T07:12:30.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-17T07:12:30.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-17T07:12:30.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.659+0000 7fafa5c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.665 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.659+0000 7f743015e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.665 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.659+0000 7fd16b915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.666+0000 7f743015e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.667+0000 7fafa5c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.667+0000 7fd16b915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.700 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2023-10-17T07:12:30.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-17T07:12:30.703 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2023-10-17T07:12:30.704 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2023-10-17T07:12:30.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-17T07:12:30.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-17T07:12:30.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.810+0000 7f58a9824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.818+0000 7f58a9824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.854+0000 7f5cf5d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.854+0000 7f473835e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.862+0000 7f5cf5d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:30.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:30.862+0000 7f473835e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2023-10-17T07:12:31.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2023-10-17T07:12:31.031 DEBUG:teuthology.parallel:result is None 2023-10-17T07:12:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2023-10-17T07:12:31.064 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2023-10-17T07:12:31.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-17T07:12:31.067 DEBUG:teuthology.parallel:result is None 2023-10-17T07:12:31.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2023-10-17T07:12:31.076 DEBUG:teuthology.parallel:result is None 2023-10-17T07:12:31.077 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-17T07:12:31.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T07:12:31.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:31.222+0000 7f6aee59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:31.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:31.228+0000 7f6aee59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:31.411 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:31.411 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2023-10-17T07:12:31.422 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-17T07:12:30.690494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82548,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490252,"statfs":{"total":322122547200,"available":322038018048,"internally_reserved":0,"allocated":1347584,"data_stored":1230629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.517509+0000","last_change":"2023-10-17T07:12:29.647861+0000","last_active":"2023-10-17T07:12:30.517509+0000","last_peered":"2023-10-17T07:12:30.517509+0000","last_clean":"2023-10-17T07:12:30.517509+0000","last_became_active":"2023-10-17T07:12:29.642197+0000","last_became_peered":"2023-10-17T07:12:29.642197+0000","last_unstale":"2023-10-17T07:12:30.517509+0000","last_undegraded":"2023-10-17T07:12:30.517509+0000","last_fullsized":"2023-10-17T07:12:30.517509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.514920+0000","last_change":"2023-10-17T07:12:29.629011+0000","last_active":"2023-10-17T07:12:30.514920+0000","last_peered":"2023-10-17T07:12:30.514920+0000","last_clean":"2023-10-17T07:12:30.514920+0000","last_became_active":"2023-10-17T07:12:29.605819+0000","last_became_peered":"2023-10-17T07:12:29.605819+0000","last_unstale":"2023-10-17T07:12:30.514920+0000","last_undegraded":"2023-10-17T07:12:30.514920+0000","last_fullsized":"2023-10-17T07:12:30.514920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.515663+0000","last_change":"2023-10-17T07:12:29.578201+0000","last_active":"2023-10-17T07:12:30.515663+0000","last_peered":"2023-10-17T07:12:30.515663+0000","last_clean":"2023-10-17T07:12:30.515663+0000","last_became_active":"2023-10-17T07:12:29.568160+0000","last_became_peered":"2023-10-17T07:12:29.568160+0000","last_unstale":"2023-10-17T07:12:30.515663+0000","last_undegraded":"2023-10-17T07:12:30.515663+0000","last_fullsized":"2023-10-17T07:12:30.515663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.518751+0000","last_change":"2023-10-17T07:12:29.603391+0000","last_active":"2023-10-17T07:12:30.518751+0000","last_peered":"2023-10-17T07:12:30.518751+0000","last_clean":"2023-10-17T07:12:30.518751+0000","last_became_active":"2023-10-17T07:12:29.598509+0000","last_became_peered":"2023-10-17T07:12:29.598509+0000","last_unstale":"2023-10-17T07:12:30.518751+0000","last_undegraded":"2023-10-17T07:12:30.518751+0000","last_fullsized":"2023-10-17T07:12:30.518751+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T07:12:29.600971+0000","last_change":"2023-10-17T07:12:29.601340+0000","last_active":"2023-10-17T07:12:29.600971+0000","last_peered":"2023-10-17T07:12:29.600971+0000","last_clean":"2023-10-17T07:12:28.496826+0000","last_became_active":"2023-10-17T07:12:29.597702+0000","last_became_peered":"2023-10-17T07:12:29.597702+0000","last_unstale":"2023-10-17T07:12:29.600971+0000","last_undegraded":"2023-10-17T07:12:29.600971+0000","last_fullsized":"2023-10-17T07:12:29.600971+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T07:12:29.640698+0000","last_change":"2023-10-17T07:12:29.641386+0000","last_active":"2023-10-17T07:12:29.640698+0000","last_peered":"2023-10-17T07:12:29.640698+0000","last_clean":"2023-10-17T07:12:28.496826+0000","last_became_active":"2023-10-17T07:12:29.632813+0000","last_became_peered":"2023-10-17T07:12:29.632813+0000","last_unstale":"2023-10-17T07:12:29.640698+0000","last_undegraded":"2023-10-17T07:12:29.640698+0000","last_fullsized":"2023-10-17T07:12:29.640698+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T07:12:29.593138+0000","last_change":"2023-10-17T07:12:29.593628+0000","last_active":"2023-10-17T07:12:29.593138+0000","last_peered":"2023-10-17T07:12:29.593138+0000","last_clean":"2023-10-17T07:12:28.496826+0000","last_became_active":"2023-10-17T07:12:29.590541+0000","last_became_peered":"2023-10-17T07:12:29.590541+0000","last_unstale":"2023-10-17T07:12:29.593138+0000","last_undegraded":"2023-10-17T07:12:29.593138+0000","last_fullsized":"2023-10-17T07:12:29.593138+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.516389+0000","last_change":"2023-10-17T07:12:29.625199+0000","last_active":"2023-10-17T07:12:30.516389+0000","last_peered":"2023-10-17T07:12:30.516389+0000","last_clean":"2023-10-17T07:12:30.516389+0000","last_became_active":"2023-10-17T07:12:29.596199+0000","last_became_peered":"2023-10-17T07:12:29.596199+0000","last_unstale":"2023-10-17T07:12:30.516389+0000","last_undegraded":"2023-10-17T07:12:30.516389+0000","last_fullsized":"2023-10-17T07:12:30.516389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.514335+0000","last_change":"2023-10-17T07:12:27.524823+0000","last_active":"2023-10-17T07:12:30.514335+0000","last_peered":"2023-10-17T07:12:30.514335+0000","last_clean":"2023-10-17T07:12:30.514335+0000","last_became_active":"2023-10-17T07:12:27.518578+0000","last_became_peered":"2023-10-17T07:12:27.518578+0000","last_unstale":"2023-10-17T07:12:30.514335+0000","last_undegraded":"2023-10-17T07:12:30.514335+0000","last_fullsized":"2023-10-17T07:12:30.514335+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:26.488530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:26.488530+0000","last_clean_scrub_stamp":"2023-10-17T07:12:26.488530+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27004,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830596,"statfs":{"total":107374182400,"available":107346530304,"internally_reserved":0,"allocated":53248,"data_stored":16631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-17T07:12:31.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T07:12:31.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:31.531+0000 7fc41b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:31.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:31.538+0000 7fc41b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:31.720 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:31.721 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2023-10-17T07:12:31.732 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-17T07:12:30.690494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82548,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490252,"statfs":{"total":322122547200,"available":322038018048,"internally_reserved":0,"allocated":1347584,"data_stored":1230629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.517509+0000","last_change":"2023-10-17T07:12:29.647861+0000","last_active":"2023-10-17T07:12:30.517509+0000","last_peered":"2023-10-17T07:12:30.517509+0000","last_clean":"2023-10-17T07:12:30.517509+0000","last_became_active":"2023-10-17T07:12:29.642197+0000","last_became_peered":"2023-10-17T07:12:29.642197+0000","last_unstale":"2023-10-17T07:12:30.517509+0000","last_undegraded":"2023-10-17T07:12:30.517509+0000","last_fullsized":"2023-10-17T07:12:30.517509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.514920+0000","last_change":"2023-10-17T07:12:29.629011+0000","last_active":"2023-10-17T07:12:30.514920+0000","last_peered":"2023-10-17T07:12:30.514920+0000","last_clean":"2023-10-17T07:12:30.514920+0000","last_became_active":"2023-10-17T07:12:29.605819+0000","last_became_peered":"2023-10-17T07:12:29.605819+0000","last_unstale":"2023-10-17T07:12:30.514920+0000","last_undegraded":"2023-10-17T07:12:30.514920+0000","last_fullsized":"2023-10-17T07:12:30.514920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.515663+0000","last_change":"2023-10-17T07:12:29.578201+0000","last_active":"2023-10-17T07:12:30.515663+0000","last_peered":"2023-10-17T07:12:30.515663+0000","last_clean":"2023-10-17T07:12:30.515663+0000","last_became_active":"2023-10-17T07:12:29.568160+0000","last_became_peered":"2023-10-17T07:12:29.568160+0000","last_unstale":"2023-10-17T07:12:30.515663+0000","last_undegraded":"2023-10-17T07:12:30.515663+0000","last_fullsized":"2023-10-17T07:12:30.515663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.518751+0000","last_change":"2023-10-17T07:12:29.603391+0000","last_active":"2023-10-17T07:12:30.518751+0000","last_peered":"2023-10-17T07:12:30.518751+0000","last_clean":"2023-10-17T07:12:30.518751+0000","last_became_active":"2023-10-17T07:12:29.598509+0000","last_became_peered":"2023-10-17T07:12:29.598509+0000","last_unstale":"2023-10-17T07:12:30.518751+0000","last_undegraded":"2023-10-17T07:12:30.518751+0000","last_fullsized":"2023-10-17T07:12:30.518751+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T07:12:29.600971+0000","last_change":"2023-10-17T07:12:29.601340+0000","last_active":"2023-10-17T07:12:29.600971+0000","last_peered":"2023-10-17T07:12:29.600971+0000","last_clean":"2023-10-17T07:12:28.496826+0000","last_became_active":"2023-10-17T07:12:29.597702+0000","last_became_peered":"2023-10-17T07:12:29.597702+0000","last_unstale":"2023-10-17T07:12:29.600971+0000","last_undegraded":"2023-10-17T07:12:29.600971+0000","last_fullsized":"2023-10-17T07:12:29.600971+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T07:12:29.640698+0000","last_change":"2023-10-17T07:12:29.641386+0000","last_active":"2023-10-17T07:12:29.640698+0000","last_peered":"2023-10-17T07:12:29.640698+0000","last_clean":"2023-10-17T07:12:28.496826+0000","last_became_active":"2023-10-17T07:12:29.632813+0000","last_became_peered":"2023-10-17T07:12:29.632813+0000","last_unstale":"2023-10-17T07:12:29.640698+0000","last_undegraded":"2023-10-17T07:12:29.640698+0000","last_fullsized":"2023-10-17T07:12:29.640698+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T07:12:29.593138+0000","last_change":"2023-10-17T07:12:29.593628+0000","last_active":"2023-10-17T07:12:29.593138+0000","last_peered":"2023-10-17T07:12:29.593138+0000","last_clean":"2023-10-17T07:12:28.496826+0000","last_became_active":"2023-10-17T07:12:29.590541+0000","last_became_peered":"2023-10-17T07:12:29.590541+0000","last_unstale":"2023-10-17T07:12:29.593138+0000","last_undegraded":"2023-10-17T07:12:29.593138+0000","last_fullsized":"2023-10-17T07:12:29.593138+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.516389+0000","last_change":"2023-10-17T07:12:29.625199+0000","last_active":"2023-10-17T07:12:30.516389+0000","last_peered":"2023-10-17T07:12:30.516389+0000","last_clean":"2023-10-17T07:12:30.516389+0000","last_became_active":"2023-10-17T07:12:29.596199+0000","last_became_peered":"2023-10-17T07:12:29.596199+0000","last_unstale":"2023-10-17T07:12:30.516389+0000","last_undegraded":"2023-10-17T07:12:30.516389+0000","last_fullsized":"2023-10-17T07:12:30.516389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:28.496826+0000","last_clean_scrub_stamp":"2023-10-17T07:12:28.496826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T07:12:30.514335+0000","last_change":"2023-10-17T07:12:27.524823+0000","last_active":"2023-10-17T07:12:30.514335+0000","last_peered":"2023-10-17T07:12:30.514335+0000","last_clean":"2023-10-17T07:12:30.514335+0000","last_became_active":"2023-10-17T07:12:27.518578+0000","last_became_peered":"2023-10-17T07:12:27.518578+0000","last_unstale":"2023-10-17T07:12:30.514335+0000","last_undegraded":"2023-10-17T07:12:30.514335+0000","last_fullsized":"2023-10-17T07:12:30.514335+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T07:12:26.488530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T07:12:26.488530+0000","last_clean_scrub_stamp":"2023-10-17T07:12:26.488530+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27004,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830596,"statfs":{"total":107374182400,"available":107346530304,"internally_reserved":0,"allocated":53248,"data_stored":16631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-17T07:12:31.733 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-17T07:12:31.733 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-17T07:12:31.733 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-17T07:12:31.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-17T07:12:31.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:31.840+0000 7fe125b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:31.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T07:12:31.845+0000 7fe125b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T07:12:32.070 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T07:12:32.071 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-17T07:12:32.083 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-17T07:12:32.083 INFO:teuthology.run_tasks:Running task workunit... 2023-10-17T07:12:32.095 INFO:tasks.workunit:Pulling workunits from ref a83d484198747052d1ed9eeed0a72e20b007e411 2023-10-17T07:12:32.096 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-17T07:12:32.096 DEBUG:teuthology.orchestra.run.smithi049:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-17T07:12:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T07:12:32.111 INFO:teuthology.orchestra.run.smithi049.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-17T07:12:32.112 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-17T07:12:32.166 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-17T07:12:32.166 DEBUG:teuthology.orchestra.run.smithi049:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-17T07:12:32.221 INFO:tasks.workunit:timeout=3h 2023-10-17T07:12:32.221 INFO:tasks.workunit:cleanup=True 2023-10-17T07:12:32.222 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a83d484198747052d1ed9eeed0a72e20b007e411 2023-10-17T07:12:32.277 INFO:tasks.workunit.client.0.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-17T07:13:21.093 INFO:tasks.workunit.client.0.smithi049.stderr:Note: switching to 'a83d484198747052d1ed9eeed0a72e20b007e411'. 2023-10-17T07:13:21.093 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.094 INFO:tasks.workunit.client.0.smithi049.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-17T07:13:21.094 INFO:tasks.workunit.client.0.smithi049.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-17T07:13:21.094 INFO:tasks.workunit.client.0.smithi049.stderr:state without impacting any branches by switching back to a branch. 2023-10-17T07:13:21.094 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.094 INFO:tasks.workunit.client.0.smithi049.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-17T07:13:21.095 INFO:tasks.workunit.client.0.smithi049.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-17T07:13:21.095 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.095 INFO:tasks.workunit.client.0.smithi049.stderr: git switch -c 2023-10-17T07:13:21.095 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.095 INFO:tasks.workunit.client.0.smithi049.stderr:Or undo this operation with: 2023-10-17T07:13:21.095 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.096 INFO:tasks.workunit.client.0.smithi049.stderr: git switch - 2023-10-17T07:13:21.096 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.096 INFO:tasks.workunit.client.0.smithi049.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-17T07:13:21.096 INFO:tasks.workunit.client.0.smithi049.stderr: 2023-10-17T07:13:21.096 INFO:tasks.workunit.client.0.smithi049.stderr:HEAD is now at a83d4841987 Merge branches sjust/wip-crimson-scrub 2023-10-17T07:13:21.098 DEBUG:teuthology.orchestra.run.smithi049:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-17T07:13:21.212 INFO:tasks.workunit.client.0.smithi049.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-17T07:13:21.215 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-17T07:13:21.215 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-17T07:13:21.924 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-17T07:13:22.031 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-17T07:13:22.074 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-17T07:13:22.075 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-17T07:13:22.075 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-17T07:13:22.116 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-17T07:13:22.119 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T07:13:22.120 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-17T07:13:22.135 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2023-10-17T07:13:22.137 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2023-10-17T07:13:22.137 DEBUG:teuthology.orchestra.run.smithi049:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a83d484198747052d1ed9eeed0a72e20b007e411 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-10-17T07:13:22.232 INFO:tasks.workunit.client.0.smithi049.stdout:seed 85587 2023-10-17T07:13:22.232 INFO:tasks.workunit.client.0.smithi049.stdout:Note: Google Test filter = TestLibRBD.* 2023-10-17T07:13:22.232 INFO:tasks.workunit.client.0.smithi049.stdout:[==========] Running 169 tests from 1 test suite. 2023-10-17T07:13:22.232 INFO:tasks.workunit.client.0.smithi049.stdout:[----------] Global test environment set-up. 2023-10-17T07:13:22.233 INFO:tasks.workunit.client.0.smithi049.stdout:[----------] 169 tests from TestLibRBD 2023-10-17T07:13:22.269 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.CreateAndStat 2023-10-17T07:13:23.870 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.041 INFO:tasks.workunit.client.0.smithi049.stdout:image has size 2097152 and order 22 2023-10-17T07:13:26.041 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.CreateAndStat (3772 ms) 2023-10-17T07:13:26.041 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2023-10-17T07:13:26.041 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.157 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (116 ms) 2023-10-17T07:13:26.157 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2023-10-17T07:13:26.158 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.253 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.CreateAndStatPP (96 ms) 2023-10-17T07:13:26.253 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.GetId 2023-10-17T07:13:26.253 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.402 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.GetId (149 ms) 2023-10-17T07:13:26.402 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.GetIdPP 2023-10-17T07:13:26.402 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.544 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.GetIdPP (142 ms) 2023-10-17T07:13:26.544 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2023-10-17T07:13:26.545 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.636 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (92 ms) 2023-10-17T07:13:26.636 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2023-10-17T07:13:26.636 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.730 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (94 ms) 2023-10-17T07:13:26.730 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2023-10-17T07:13:26.730 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.819 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (89 ms) 2023-10-17T07:13:26.819 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2023-10-17T07:13:26.820 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:26.914 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (95 ms) 2023-10-17T07:13:26.914 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.OpenAio 2023-10-17T07:13:26.915 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:27.006 INFO:tasks.workunit.client.0.smithi049.stdout:image has size 2097152 and order 22 2023-10-17T07:13:27.006 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.OpenAio (92 ms) 2023-10-17T07:13:27.007 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.OpenAioFail 2023-10-17T07:13:27.013 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.OpenAioFail (7 ms) 2023-10-17T07:13:27.014 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.OpenAioPP 2023-10-17T07:13:27.014 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:27.156 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.OpenAioPP (143 ms) 2023-10-17T07:13:27.156 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2023-10-17T07:13:27.163 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.OpenAioFailPP (7 ms) 2023-10-17T07:13:27.163 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2023-10-17T07:13:27.163 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:27.409 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.ResizeAndStat (246 ms) 2023-10-17T07:13:27.410 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2023-10-17T07:13:27.410 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:27.619 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (210 ms) 2023-10-17T07:13:27.619 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2023-10-17T07:13:27.619 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:27.849 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (230 ms) 2023-10-17T07:13:27.849 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2023-10-17T07:13:27.849 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:28.084 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (235 ms) 2023-10-17T07:13:28.084 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2023-10-17T07:13:30.171 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:32.150 INFO:tasks.workunit.client.0.smithi049.stdout:image: image18 2023-10-17T07:13:32.150 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image18 2023-10-17T07:13:32.150 INFO:tasks.workunit.client.0.smithi049.stdout:found image18 2023-10-17T07:13:32.151 INFO:tasks.workunit.client.0.smithi049.stdout:erased image18 2023-10-17T07:13:32.151 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:32.671 INFO:tasks.workunit.client.0.smithi049.stdout:image: image18 2023-10-17T07:13:32.671 INFO:tasks.workunit.client.0.smithi049.stdout:image: image19 2023-10-17T07:13:32.671 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image18 2023-10-17T07:13:32.671 INFO:tasks.workunit.client.0.smithi049.stdout:found image18 2023-10-17T07:13:32.672 INFO:tasks.workunit.client.0.smithi049.stdout:erased image18 2023-10-17T07:13:32.672 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image19 2023-10-17T07:13:32.672 INFO:tasks.workunit.client.0.smithi049.stdout:found image19 2023-10-17T07:13:32.672 INFO:tasks.workunit.client.0.smithi049.stdout:erased image19 2023-10-17T07:13:32.672 INFO:tasks.workunit.client.0.smithi049.stdout:image: image19 2023-10-17T07:13:32.672 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image19 2023-10-17T07:13:32.673 INFO:tasks.workunit.client.0.smithi049.stdout:found image19 2023-10-17T07:13:32.673 INFO:tasks.workunit.client.0.smithi049.stdout:erased image19 2023-10-17T07:13:32.673 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (4587 ms) 2023-10-17T07:13:32.673 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2023-10-17T07:13:33.908 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:36.120 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 1 2023-10-17T07:13:36.120 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:13:36.121 INFO:tasks.workunit.client.0.smithi049.stdout:image: image20 2023-10-17T07:13:36.121 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image20 2023-10-17T07:13:36.121 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:36.167 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 2 2023-10-17T07:13:36.167 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:13:36.167 INFO:tasks.workunit.client.0.smithi049.stdout:image: image20 2023-10-17T07:13:36.168 INFO:tasks.workunit.client.0.smithi049.stdout:image: image21 2023-10-17T07:13:36.168 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image20 2023-10-17T07:13:36.168 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image21 2023-10-17T07:13:36.534 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 1 2023-10-17T07:13:36.534 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:13:36.534 INFO:tasks.workunit.client.0.smithi049.stdout:image: image21 2023-10-17T07:13:36.534 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image21 2023-10-17T07:13:36.535 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3863 ms) 2023-10-17T07:13:36.535 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCopy 2023-10-17T07:13:37.924 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:43.217 INFO:tasks.workunit.client.0.smithi049.stdout:image: image22 2023-10-17T07:13:43.217 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image22 2023-10-17T07:13:43.217 INFO:tasks.workunit.client.0.smithi049.stdout:found image22 2023-10-17T07:13:43.218 INFO:tasks.workunit.client.0.smithi049.stdout:erased image22 2023-10-17T07:13:43.218 INFO:tasks.workunit.client.0.smithi049.stdout:image: image22 2023-10-17T07:13:43.218 INFO:tasks.workunit.client.0.smithi049.stdout:image: image23 2023-10-17T07:13:43.218 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image22 2023-10-17T07:13:43.218 INFO:tasks.workunit.client.0.smithi049.stdout:found image22 2023-10-17T07:13:43.219 INFO:tasks.workunit.client.0.smithi049.stdout:erased image22 2023-10-17T07:13:43.219 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image23 2023-10-17T07:13:43.219 INFO:tasks.workunit.client.0.smithi049.stdout:found image23 2023-10-17T07:13:43.219 INFO:tasks.workunit.client.0.smithi049.stdout:erased image23 2023-10-17T07:13:43.219 INFO:tasks.workunit.client.0.smithi049.stdout:100.00% done 2023-10-17T07:13:43.220 INFO:tasks.workunit.client.0.smithi049.stdout:image: image22 2023-10-17T07:13:43.220 INFO:tasks.workunit.client.0.smithi049.stdout:image: image23 2023-10-17T07:13:43.220 INFO:tasks.workunit.client.0.smithi049.stdout:image: image24 2023-10-17T07:13:43.220 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image22 2023-10-17T07:13:43.220 INFO:tasks.workunit.client.0.smithi049.stdout:found image22 2023-10-17T07:13:43.220 INFO:tasks.workunit.client.0.smithi049.stdout:erased image22 2023-10-17T07:13:43.221 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image23 2023-10-17T07:13:43.221 INFO:tasks.workunit.client.0.smithi049.stdout:found image23 2023-10-17T07:13:43.221 INFO:tasks.workunit.client.0.smithi049.stdout:erased image23 2023-10-17T07:13:43.221 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image24 2023-10-17T07:13:43.221 INFO:tasks.workunit.client.0.smithi049.stdout:found image24 2023-10-17T07:13:43.222 INFO:tasks.workunit.client.0.smithi049.stdout:erased image24 2023-10-17T07:13:43.222 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCopy (6683 ms) 2023-10-17T07:13:43.222 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCopyPP 2023-10-17T07:13:44.945 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:13:49.457 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 1 2023-10-17T07:13:49.457 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:13:49.457 INFO:tasks.workunit.client.0.smithi049.stdout:image: image25 2023-10-17T07:13:49.457 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image25 2023-10-17T07:13:49.571 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 2 2023-10-17T07:13:49.572 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:13:49.572 INFO:tasks.workunit.client.0.smithi049.stdout:image: image25 2023-10-17T07:13:49.572 INFO:tasks.workunit.client.0.smithi049.stdout:image: image26 2023-10-17T07:13:49.572 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image25 2023-10-17T07:13:49.572 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image26 2023-10-17T07:13:49.934 INFO:tasks.workunit.client.0.smithi049.stdout:100.00% done 2023-10-17T07:13:49.934 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 3 2023-10-17T07:13:49.935 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 3 2023-10-17T07:13:49.935 INFO:tasks.workunit.client.0.smithi049.stdout:image: image25 2023-10-17T07:13:49.935 INFO:tasks.workunit.client.0.smithi049.stdout:image: image26 2023-10-17T07:13:49.935 INFO:tasks.workunit.client.0.smithi049.stdout:image: image27 2023-10-17T07:13:49.936 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image25 2023-10-17T07:13:49.936 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image26 2023-10-17T07:13:49.936 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image27 2023-10-17T07:13:50.244 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCopyPP (7027 ms) 2023-10-17T07:13:50.245 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2023-10-17T07:13:51.962 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:02.879 INFO:tasks.workunit.client.0.smithi049.stdout:image: image28 2023-10-17T07:14:02.880 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image28 2023-10-17T07:14:02.880 INFO:tasks.workunit.client.0.smithi049.stdout:found image28 2023-10-17T07:14:02.880 INFO:tasks.workunit.client.0.smithi049.stdout:erased image28 2023-10-17T07:14:02.881 INFO:tasks.workunit.client.0.smithi049.stdout:image: image28 2023-10-17T07:14:02.881 INFO:tasks.workunit.client.0.smithi049.stdout:image: image29 2023-10-17T07:14:02.881 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image28 2023-10-17T07:14:02.881 INFO:tasks.workunit.client.0.smithi049.stdout:found image28 2023-10-17T07:14:02.881 INFO:tasks.workunit.client.0.smithi049.stdout:erased image28 2023-10-17T07:14:02.881 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image29 2023-10-17T07:14:02.882 INFO:tasks.workunit.client.0.smithi049.stdout:found image29 2023-10-17T07:14:02.882 INFO:tasks.workunit.client.0.smithi049.stdout:erased image29 2023-10-17T07:14:02.882 INFO:tasks.workunit.client.0.smithi049.stdout:100.00% done 2023-10-17T07:14:02.882 INFO:tasks.workunit.client.0.smithi049.stdout:image: image28 2023-10-17T07:14:02.882 INFO:tasks.workunit.client.0.smithi049.stdout:image: image29 2023-10-17T07:14:02.883 INFO:tasks.workunit.client.0.smithi049.stdout:image: image30 2023-10-17T07:14:02.883 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image28 2023-10-17T07:14:02.883 INFO:tasks.workunit.client.0.smithi049.stdout:found image28 2023-10-17T07:14:02.883 INFO:tasks.workunit.client.0.smithi049.stdout:erased image28 2023-10-17T07:14:02.883 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image29 2023-10-17T07:14:02.884 INFO:tasks.workunit.client.0.smithi049.stdout:found image29 2023-10-17T07:14:02.884 INFO:tasks.workunit.client.0.smithi049.stdout:erased image29 2023-10-17T07:14:02.884 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image30 2023-10-17T07:14:02.884 INFO:tasks.workunit.client.0.smithi049.stdout:found image30 2023-10-17T07:14:02.884 INFO:tasks.workunit.client.0.smithi049.stdout:erased image30 2023-10-17T07:14:02.884 INFO:tasks.workunit.client.0.smithi049.stdout:image: image28 2023-10-17T07:14:02.885 INFO:tasks.workunit.client.0.smithi049.stdout:image: image29 2023-10-17T07:14:02.885 INFO:tasks.workunit.client.0.smithi049.stdout:image: image30 2023-10-17T07:14:02.885 INFO:tasks.workunit.client.0.smithi049.stdout:image: image31 2023-10-17T07:14:02.885 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image28 2023-10-17T07:14:02.885 INFO:tasks.workunit.client.0.smithi049.stdout:found image28 2023-10-17T07:14:02.886 INFO:tasks.workunit.client.0.smithi049.stdout:erased image28 2023-10-17T07:14:02.886 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image29 2023-10-17T07:14:02.886 INFO:tasks.workunit.client.0.smithi049.stdout:found image29 2023-10-17T07:14:02.886 INFO:tasks.workunit.client.0.smithi049.stdout:erased image29 2023-10-17T07:14:02.886 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image30 2023-10-17T07:14:02.886 INFO:tasks.workunit.client.0.smithi049.stdout:found image30 2023-10-17T07:14:02.887 INFO:tasks.workunit.client.0.smithi049.stdout:erased image30 2023-10-17T07:14:02.887 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image31 2023-10-17T07:14:02.887 INFO:tasks.workunit.client.0.smithi049.stdout:found image31 2023-10-17T07:14:02.887 INFO:tasks.workunit.client.0.smithi049.stdout:erased image31 2023-10-17T07:14:02.887 INFO:tasks.workunit.client.0.smithi049.stdout:image: image28 2023-10-17T07:14:02.887 INFO:tasks.workunit.client.0.smithi049.stdout:image: image29 2023-10-17T07:14:02.888 INFO:tasks.workunit.client.0.smithi049.stdout:image: image30 2023-10-17T07:14:02.888 INFO:tasks.workunit.client.0.smithi049.stdout:image: image31 2023-10-17T07:14:02.888 INFO:tasks.workunit.client.0.smithi049.stdout:image: image32 2023-10-17T07:14:02.888 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image28 2023-10-17T07:14:02.888 INFO:tasks.workunit.client.0.smithi049.stdout:found image28 2023-10-17T07:14:02.889 INFO:tasks.workunit.client.0.smithi049.stdout:erased image28 2023-10-17T07:14:02.889 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image29 2023-10-17T07:14:02.889 INFO:tasks.workunit.client.0.smithi049.stdout:found image29 2023-10-17T07:14:02.889 INFO:tasks.workunit.client.0.smithi049.stdout:erased image29 2023-10-17T07:14:02.889 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image30 2023-10-17T07:14:02.889 INFO:tasks.workunit.client.0.smithi049.stdout:found image30 2023-10-17T07:14:02.890 INFO:tasks.workunit.client.0.smithi049.stdout:erased image30 2023-10-17T07:14:02.890 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image31 2023-10-17T07:14:02.890 INFO:tasks.workunit.client.0.smithi049.stdout:found image31 2023-10-17T07:14:02.890 INFO:tasks.workunit.client.0.smithi049.stdout:erased image31 2023-10-17T07:14:02.890 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image32 2023-10-17T07:14:02.890 INFO:tasks.workunit.client.0.smithi049.stdout:found image32 2023-10-17T07:14:02.891 INFO:tasks.workunit.client.0.smithi049.stdout:erased image32 2023-10-17T07:14:02.891 INFO:tasks.workunit.client.0.smithi049.stdout:100.00% done 2023-10-17T07:14:02.891 INFO:tasks.workunit.client.0.smithi049.stdout:image: image28 2023-10-17T07:14:02.891 INFO:tasks.workunit.client.0.smithi049.stdout:image: image29 2023-10-17T07:14:02.891 INFO:tasks.workunit.client.0.smithi049.stdout:image: image30 2023-10-17T07:14:02.892 INFO:tasks.workunit.client.0.smithi049.stdout:image: image31 2023-10-17T07:14:02.892 INFO:tasks.workunit.client.0.smithi049.stdout:image: image32 2023-10-17T07:14:02.892 INFO:tasks.workunit.client.0.smithi049.stdout:image: image33 2023-10-17T07:14:02.892 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image28 2023-10-17T07:14:02.892 INFO:tasks.workunit.client.0.smithi049.stdout:found image28 2023-10-17T07:14:02.892 INFO:tasks.workunit.client.0.smithi049.stdout:erased image28 2023-10-17T07:14:02.893 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image29 2023-10-17T07:14:02.893 INFO:tasks.workunit.client.0.smithi049.stdout:found image29 2023-10-17T07:14:02.893 INFO:tasks.workunit.client.0.smithi049.stdout:erased image29 2023-10-17T07:14:02.893 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image30 2023-10-17T07:14:02.893 INFO:tasks.workunit.client.0.smithi049.stdout:found image30 2023-10-17T07:14:02.894 INFO:tasks.workunit.client.0.smithi049.stdout:erased image30 2023-10-17T07:14:02.894 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image31 2023-10-17T07:14:02.894 INFO:tasks.workunit.client.0.smithi049.stdout:found image31 2023-10-17T07:14:02.894 INFO:tasks.workunit.client.0.smithi049.stdout:erased image31 2023-10-17T07:14:02.894 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image32 2023-10-17T07:14:02.895 INFO:tasks.workunit.client.0.smithi049.stdout:found image32 2023-10-17T07:14:02.895 INFO:tasks.workunit.client.0.smithi049.stdout:erased image32 2023-10-17T07:14:02.895 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image33 2023-10-17T07:14:02.895 INFO:tasks.workunit.client.0.smithi049.stdout:found image33 2023-10-17T07:14:02.895 INFO:tasks.workunit.client.0.smithi049.stdout:erased image33 2023-10-17T07:14:02.895 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestDeepCopy (12635 ms) 2023-10-17T07:14:02.896 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2023-10-17T07:14:04.850 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:09.694 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 1 2023-10-17T07:14:09.695 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:09.695 INFO:tasks.workunit.client.0.smithi049.stdout:image: image34 2023-10-17T07:14:09.695 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image34 2023-10-17T07:14:09.861 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 2 2023-10-17T07:14:09.862 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:14:09.862 INFO:tasks.workunit.client.0.smithi049.stdout:image: image34 2023-10-17T07:14:09.862 INFO:tasks.workunit.client.0.smithi049.stdout:image: image35 2023-10-17T07:14:09.862 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image34 2023-10-17T07:14:09.862 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image35 2023-10-17T07:14:10.419 INFO:tasks.workunit.client.0.smithi049.stdout:100.00% done 2023-10-17T07:14:10.419 INFO:tasks.workunit.client.0.smithi049.stdout:num images is: 3 2023-10-17T07:14:10.419 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 3 2023-10-17T07:14:10.420 INFO:tasks.workunit.client.0.smithi049.stdout:image: image34 2023-10-17T07:14:10.420 INFO:tasks.workunit.client.0.smithi049.stdout:image: image35 2023-10-17T07:14:10.420 INFO:tasks.workunit.client.0.smithi049.stdout:image: image36 2023-10-17T07:14:10.420 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image34 2023-10-17T07:14:10.420 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image35 2023-10-17T07:14:10.421 INFO:tasks.workunit.client.0.smithi049.stdout:expected = image36 2023-10-17T07:14:10.712 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (7832 ms) 2023-10-17T07:14:10.712 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2023-10-17T07:14:10.712 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:14.469 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 1 2023-10-17T07:14:14.469 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:14.470 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1 2023-10-17T07:14:14.470 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1 with size 2097152 2023-10-17T07:14:14.470 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 2 2023-10-17T07:14:14.470 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:14:14.470 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1 2023-10-17T07:14:14.471 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2 2023-10-17T07:14:14.471 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1 with size 2097152 2023-10-17T07:14:14.471 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2 with size 4194304 2023-10-17T07:14:14.471 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 1 2023-10-17T07:14:14.471 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:14.471 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2 2023-10-17T07:14:14.472 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2 with size 4194304 2023-10-17T07:14:14.472 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 0 2023-10-17T07:14:14.472 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 0 2023-10-17T07:14:14.472 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3758 ms) 2023-10-17T07:14:14.472 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2023-10-17T07:14:14.473 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:16.522 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (2053 ms) 2023-10-17T07:14:16.523 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2023-10-17T07:14:16.523 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:17.437 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 1 2023-10-17T07:14:17.438 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:17.438 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1 2023-10-17T07:14:17.439 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1 with size 2097152 2023-10-17T07:14:18.566 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 2 2023-10-17T07:14:18.566 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:14:18.566 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1 2023-10-17T07:14:18.567 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2 2023-10-17T07:14:18.567 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1 with size 2097152 2023-10-17T07:14:18.567 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2 with size 4194304 2023-10-17T07:14:19.388 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 1 2023-10-17T07:14:19.389 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:19.389 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2 2023-10-17T07:14:19.389 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2 with size 4194304 2023-10-17T07:14:20.466 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 0 2023-10-17T07:14:20.467 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 0 2023-10-17T07:14:20.492 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (3970 ms) 2023-10-17T07:14:20.492 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2023-10-17T07:14:20.493 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:26.518 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 0 2023-10-17T07:14:26.519 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 0 2023-10-17T07:14:26.629 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6137 ms) 2023-10-17T07:14:26.630 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2023-10-17T07:14:26.630 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:27.414 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 1 2023-10-17T07:14:27.414 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:27.414 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1 2023-10-17T07:14:27.415 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1 with size 2097152 2023-10-17T07:14:28.495 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 2 2023-10-17T07:14:28.495 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:14:28.495 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1 2023-10-17T07:14:28.495 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2 2023-10-17T07:14:28.496 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1 with size 2097152 2023-10-17T07:14:28.496 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2 with size 4194304 2023-10-17T07:14:28.534 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 2 2023-10-17T07:14:28.534 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 2 2023-10-17T07:14:28.535 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap1-rename 2023-10-17T07:14:28.535 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2 2023-10-17T07:14:28.535 INFO:tasks.workunit.client.0.smithi049.stdout:found snap1-rename with size 2097152 2023-10-17T07:14:28.535 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2 with size 4194304 2023-10-17T07:14:29.570 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 1 2023-10-17T07:14:29.571 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 1 2023-10-17T07:14:29.571 INFO:tasks.workunit.client.0.smithi049.stdout:snap: snap2-rename 2023-10-17T07:14:29.571 INFO:tasks.workunit.client.0.smithi049.stdout:found snap2-rename with size 4194304 2023-10-17T07:14:30.466 INFO:tasks.workunit.client.0.smithi049.stdout:num snaps is: 0 2023-10-17T07:14:30.466 INFO:tasks.workunit.client.0.smithi049.stdout:expected: 0 2023-10-17T07:14:30.520 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3891 ms) 2023-10-17T07:14:30.520 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2023-10-17T07:14:32.092 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:32.092 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:32.092 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:32.093 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:37.015 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (6495 ms) 2023-10-17T07:14:37.015 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2023-10-17T07:14:39.574 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:41.719 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:14:59.038 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.RemoveFullTry (22023 ms) 2023-10-17T07:14:59.039 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2023-10-17T07:15:03.472 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:15:23.816 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (24778 ms) 2023-10-17T07:15:23.817 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestIO 2023-10-17T07:15:23.817 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:15:31.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-17T07:15:36.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2023-10-17T07:15:42.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2023-10-17T07:15:47.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2023-10-17T07:15:53.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2023-10-17T07:15:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2023-10-17T07:16:03.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2023-10-17T07:16:05.348 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:05.348 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:05.349 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:05.349 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:05.349 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:05.349 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.349 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:05.350 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.350 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.350 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:05.350 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.350 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:05.351 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.351 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.351 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:05.351 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.351 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:05.351 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.352 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.352 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:05.352 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.352 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:05.352 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.353 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.353 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:05.353 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.353 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:05.353 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.353 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.354 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:05.354 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:05.354 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:05.354 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:05.354 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:05.355 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:05.355 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:05.355 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:05.355 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:05.355 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:05.355 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:05.356 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.356 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:05.356 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.356 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.356 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:05.357 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.357 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:05.357 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.357 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.357 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:05.357 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.358 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:05.358 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.358 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.358 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:05.358 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.359 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:05.359 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.359 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.359 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:05.359 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.359 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:05.360 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.360 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.360 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:05.360 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.360 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.360 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.361 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.361 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.361 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.361 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:05.361 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:05.362 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:05.362 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.362 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:05.362 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:05.362 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:05.362 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.363 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:05.363 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:05.363 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:05.363 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.363 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:05.364 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:05.364 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:05.364 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.364 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:05.364 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:05.364 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:05.365 INFO:tasks.workunit.client.0.smithi049.stdout:discard: 512~512 = 512 2023-10-17T07:16:05.365 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.365 INFO:tasks.workunit.client.0.smithi049.stdout:aio discard: 1536~512 = 0 2023-10-17T07:16:05.365 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.365 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.366 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.366 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.366 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:05.366 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 0 2023-10-17T07:16:05.366 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.366 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.367 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 0 2023-10-17T07:16:05.367 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.367 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.367 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 16384 2023-10-17T07:16:05.367 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.367 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.368 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 16384 2023-10-17T07:16:05.368 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.368 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.368 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.368 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.369 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 49152 2023-10-17T07:16:05.369 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.369 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.369 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 49152 2023-10-17T07:16:05.369 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.369 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.370 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 65536 2023-10-17T07:16:05.370 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.370 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.370 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 65536 2023-10-17T07:16:05.370 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.371 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.371 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.371 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.371 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 98304 2023-10-17T07:16:05.371 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.371 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.372 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 98304 2023-10-17T07:16:05.372 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.372 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.372 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 114688 2023-10-17T07:16:05.372 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.373 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.373 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 114688 2023-10-17T07:16:05.373 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.373 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.373 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.373 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.374 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 147456 2023-10-17T07:16:05.374 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.374 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.374 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 147456 2023-10-17T07:16:05.374 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.375 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.375 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 163840 2023-10-17T07:16:05.375 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.375 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.375 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 163840 2023-10-17T07:16:05.375 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.376 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.376 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.376 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.376 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 196608 2023-10-17T07:16:05.376 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.376 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.377 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 196608 2023-10-17T07:16:05.377 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.377 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.377 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 212992 2023-10-17T07:16:05.377 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.378 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.378 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 212992 2023-10-17T07:16:05.378 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.378 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.378 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.379 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.379 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.379 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.379 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.379 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.379 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:05.380 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.380 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.380 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.380 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.380 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.381 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.381 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:05.381 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.381 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.381 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.381 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.382 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.382 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.382 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:05.382 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.382 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.383 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.383 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.383 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.383 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.383 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:05.383 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.384 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.384 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.384 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.384 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.384 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.384 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.385 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:05.385 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.385 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.385 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.385 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.386 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:05.386 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.386 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.386 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:05.386 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:05.386 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:05.387 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:05.387 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:05.387 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:05.387 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:05.387 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:09.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2023-10-17T07:16:13.930 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.930 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.931 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.931 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.931 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.931 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.931 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.932 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.932 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.932 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.932 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.932 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.932 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.933 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.933 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.933 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.933 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.933 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.933 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.934 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.934 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.934 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.934 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.934 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.935 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.935 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.935 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.935 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.935 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.935 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.936 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.936 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.936 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.936 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.936 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.936 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.937 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.937 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.937 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.937 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.937 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.937 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.938 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.938 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.938 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.938 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.938 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.938 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.939 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.939 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.939 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.939 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.939 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.939 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.940 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.940 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.940 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.940 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.940 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.940 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.941 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.941 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.941 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.941 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.941 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.941 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.942 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.942 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.942 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.942 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.942 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.943 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.943 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.943 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.943 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.943 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.943 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.944 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.944 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.944 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.944 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.944 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.944 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.945 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.946 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.946 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.946 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.946 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.946 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.947 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.947 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.947 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.947 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.947 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.948 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.948 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.948 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.948 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.948 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.948 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.949 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.949 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.949 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.949 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.949 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:13.950 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:13.950 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:13.950 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:13.950 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:13.950 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.951 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.951 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.951 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:13.951 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:13.951 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestIO (50114 ms) 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2023-10-17T07:16:13.952 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:16:14.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2023-10-17T07:16:20.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2023-10-17T07:16:25.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2023-10-17T07:16:28.022 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:28.022 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:28.023 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:28.023 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:28.023 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:28.023 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.023 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:28.024 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.024 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.024 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:28.024 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.024 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:28.024 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.025 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.025 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:28.025 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.025 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:28.025 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.026 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.026 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:28.026 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.026 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:28.026 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.027 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.027 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:28.027 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.027 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:28.027 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.027 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.028 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:28.028 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:28.028 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:28.028 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:28.028 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:28.029 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:28.029 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:28.029 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:28.029 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:28.029 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:28.029 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:28.030 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.030 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:28.030 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.030 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.030 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:28.031 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.031 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:28.031 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.031 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.031 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:28.032 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.032 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:28.032 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.032 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.032 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:28.032 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.033 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:28.033 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.033 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.033 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:28.033 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.034 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:28.034 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.034 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.034 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:28.034 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.034 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.035 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.035 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.035 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.035 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.035 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:28.036 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:28.036 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:28.036 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.036 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:28.036 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:28.036 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:28.037 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.037 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:28.037 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:28.037 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:28.038 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.038 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:28.038 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:28.038 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:28.038 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.039 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:28.039 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:28.039 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:28.039 INFO:tasks.workunit.client.0.smithi049.stdout:discard: 512~512 = 512 2023-10-17T07:16:28.039 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.040 INFO:tasks.workunit.client.0.smithi049.stdout:aio discard: 1536~512 = 0 2023-10-17T07:16:28.040 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.040 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.041 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.041 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.041 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:28.041 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 0 2023-10-17T07:16:28.041 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.042 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.042 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 0 2023-10-17T07:16:28.042 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.042 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.042 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 16384 2023-10-17T07:16:28.043 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.043 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.043 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 16384 2023-10-17T07:16:28.043 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.043 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.043 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.044 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.044 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 49152 2023-10-17T07:16:28.044 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.044 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.044 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 49152 2023-10-17T07:16:28.045 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.045 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.045 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 65536 2023-10-17T07:16:28.045 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.045 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.045 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 65536 2023-10-17T07:16:28.046 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.046 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.046 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.046 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.046 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 98304 2023-10-17T07:16:28.047 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.047 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.047 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 98304 2023-10-17T07:16:28.047 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.047 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.047 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 114688 2023-10-17T07:16:28.048 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.048 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.048 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 114688 2023-10-17T07:16:28.048 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.048 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.048 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.049 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.049 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 147456 2023-10-17T07:16:28.049 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.049 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.049 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 147456 2023-10-17T07:16:28.050 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.050 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.050 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 163840 2023-10-17T07:16:28.050 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.050 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.051 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 163840 2023-10-17T07:16:28.051 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.051 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.051 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.051 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.051 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 196608 2023-10-17T07:16:28.052 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.052 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.052 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 196608 2023-10-17T07:16:28.052 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.052 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.053 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 212992 2023-10-17T07:16:28.053 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.053 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.053 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 212992 2023-10-17T07:16:28.053 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.053 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.054 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.054 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.054 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.054 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.054 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.055 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.055 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:28.055 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.055 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.055 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.056 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.056 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.056 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.056 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:28.056 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.057 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.057 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.057 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.057 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.057 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.057 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:28.058 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.058 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.058 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.058 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.058 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.059 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.059 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:28.059 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.059 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.059 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.060 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.060 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.060 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.060 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.060 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:28.061 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.061 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.061 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.061 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.061 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:28.062 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.062 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.062 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:28.062 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:28.062 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:28.063 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:28.063 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:28.063 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:28.063 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:28.063 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:30.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~59s 2023-10-17T07:16:36.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2023-10-17T07:16:40.959 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.959 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.960 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.960 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.960 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.960 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.961 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.961 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.961 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.961 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.961 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.961 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.962 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.962 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.962 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.962 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.963 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.963 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.963 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.963 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.963 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.963 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.964 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.964 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.964 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.964 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.964 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.965 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.965 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.965 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.965 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.965 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.966 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.966 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.966 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.966 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.966 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.967 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.967 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.967 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.967 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.967 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.967 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.968 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.968 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.968 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.968 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.968 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.969 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.970 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.970 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.970 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.970 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.970 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.971 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.971 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.971 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.971 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.971 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.971 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.972 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.972 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.972 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.972 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.972 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.972 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.973 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.973 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.973 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.973 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.973 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.973 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.974 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.974 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.974 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.974 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.974 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.975 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.975 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.975 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.975 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.975 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.975 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.976 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.976 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.976 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.976 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.976 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.977 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.977 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.977 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.977 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.977 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.977 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.978 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.978 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.978 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.978 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.978 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.978 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.979 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.979 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.979 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.979 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.980 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:40.980 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:40.980 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:40.980 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:40.980 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:40.981 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.981 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.981 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.981 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:40.981 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:40.982 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:40.982 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:40.982 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:40.982 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:40.982 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:40.983 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 4 2023-10-17T07:16:40.983 INFO:tasks.workunit.client.0.smithi049.stdout:read: 4 2023-10-17T07:16:40.983 INFO:tasks.workunit.client.0.smithi049.stdout:read: 4 2023-10-17T07:16:40.983 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (27029 ms) 2023-10-17T07:16:40.983 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2023-10-17T07:16:40.983 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:16:41.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2023-10-17T07:16:47.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~76s 2023-10-17T07:16:52.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2023-10-17T07:16:58.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2023-10-17T07:16:59.662 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:59.663 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:59.663 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:59.663 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:59.663 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:16:59.663 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.664 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:59.664 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.664 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.664 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:59.664 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.664 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:59.665 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.665 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.665 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:59.665 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.665 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:59.666 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.666 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.666 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:59.666 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.666 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:59.667 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.667 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.667 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:59.667 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.667 INFO:tasks.workunit.client.0.smithi049.stdout:started write 2023-10-17T07:16:59.667 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.668 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.668 INFO:tasks.workunit.client.0.smithi049.stdout:finished write 2023-10-17T07:16:59.668 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:59.668 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:59.668 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:59.669 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:59.669 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:59.669 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:59.669 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:59.669 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:59.669 INFO:tasks.workunit.client.0.smithi049.stdout:start compare and write 2023-10-17T07:16:59.670 INFO:tasks.workunit.client.0.smithi049.stdout:compare and wrote: 512 2023-10-17T07:16:59.670 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.670 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:59.670 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.671 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.671 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:59.671 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.671 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:59.671 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.671 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.672 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:59.672 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.672 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:59.672 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.672 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.673 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:59.673 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.673 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:59.673 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.673 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.673 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:59.674 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.674 INFO:tasks.workunit.client.0.smithi049.stdout:started aio compare and write 2023-10-17T07:16:59.674 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.674 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.674 INFO:tasks.workunit.client.0.smithi049.stdout:finished aio compare and write 2023-10-17T07:16:59.675 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.675 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.675 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.675 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.675 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.675 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.676 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:59.676 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:59.676 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:59.676 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.676 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:59.676 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:59.677 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:59.677 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.677 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:59.677 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:59.677 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:59.678 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.678 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:59.678 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:59.678 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:59.678 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.679 INFO:tasks.workunit.client.0.smithi049.stdout:started read 2023-10-17T07:16:59.679 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:16:59.679 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 512 2023-10-17T07:16:59.679 INFO:tasks.workunit.client.0.smithi049.stdout:discard: 512~512 = 512 2023-10-17T07:16:59.679 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.679 INFO:tasks.workunit.client.0.smithi049.stdout:aio discard: 1536~512 = 0 2023-10-17T07:16:59.680 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.680 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.680 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.680 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.680 INFO:tasks.workunit.client.0.smithi049.stdout:read: 512 2023-10-17T07:16:59.681 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 0 2023-10-17T07:16:59.681 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.681 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.681 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 0 2023-10-17T07:16:59.681 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.681 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.682 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 16384 2023-10-17T07:16:59.682 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.682 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.682 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 16384 2023-10-17T07:16:59.682 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.683 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.683 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.683 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.683 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 49152 2023-10-17T07:16:59.683 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.683 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.684 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 49152 2023-10-17T07:16:59.684 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.684 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.684 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 65536 2023-10-17T07:16:59.684 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.684 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.685 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 65536 2023-10-17T07:16:59.685 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.685 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.685 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.685 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.685 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 98304 2023-10-17T07:16:59.686 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.686 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.686 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 98304 2023-10-17T07:16:59.686 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.686 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.687 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 114688 2023-10-17T07:16:59.687 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.687 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.687 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 114688 2023-10-17T07:16:59.687 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.687 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.688 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.688 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.688 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 147456 2023-10-17T07:16:59.688 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.688 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.689 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 147456 2023-10-17T07:16:59.689 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.689 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.689 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 163840 2023-10-17T07:16:59.689 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.690 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.690 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 163840 2023-10-17T07:16:59.690 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.690 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.690 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.690 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.691 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 196608 2023-10-17T07:16:59.691 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.691 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.691 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 196608 2023-10-17T07:16:59.691 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.692 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.692 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 212992 2023-10-17T07:16:59.692 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.692 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.692 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 212992 2023-10-17T07:16:59.692 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.693 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.693 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.693 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.693 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.693 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.694 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.694 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.694 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:59.694 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.694 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.695 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.695 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.695 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.695 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.695 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:59.695 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.696 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.696 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.696 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.696 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.696 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.696 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:59.697 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.697 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.697 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.697 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.697 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.697 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.698 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:59.698 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.698 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.698 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.698 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.699 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.699 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.699 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.699 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:16:59.699 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.699 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.700 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.700 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.700 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:59.700 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.700 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.701 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:16:59.701 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:16:59.701 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:16:59.701 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:16:59.701 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:16:59.701 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:16:59.702 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:16:59.702 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:03.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2023-10-17T07:17:08.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~97s 2023-10-17T07:17:12.910 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.911 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.911 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.911 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.911 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.912 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.912 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.912 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.912 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.912 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.913 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.913 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.913 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.913 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.913 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.914 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.914 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.914 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.914 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.914 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.915 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.915 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.915 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.915 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.915 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.916 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.916 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.916 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.916 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.916 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.917 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.917 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.917 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.917 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.917 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.918 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.918 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.918 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.918 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.918 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.919 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.919 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.919 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.919 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.919 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.919 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.920 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.920 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.920 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.920 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.920 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.921 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.921 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.921 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.921 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.921 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.922 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.922 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.922 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.922 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.922 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.923 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.923 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.923 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.923 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.923 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.924 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.924 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.924 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.924 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.924 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.924 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.925 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.925 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.925 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.925 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.925 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.926 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.926 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.926 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.926 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.926 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.927 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.927 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.927 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.927 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.927 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.927 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.928 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.928 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.928 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.928 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.930 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.930 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.930 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.931 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.931 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.931 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.931 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.931 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.932 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.932 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.932 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.932 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.932 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.932 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.933 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.933 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.933 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.933 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.933 INFO:tasks.workunit.client.0.smithi049.stdout:write completion cb called! 2023-10-17T07:17:12.934 INFO:tasks.workunit.client.0.smithi049.stdout:return value is: 0 2023-10-17T07:17:12.934 INFO:tasks.workunit.client.0.smithi049.stdout:finished writesame 2023-10-17T07:17:12.934 INFO:tasks.workunit.client.0.smithi049.stdout:to verify the data 2023-10-17T07:17:12.934 INFO:tasks.workunit.client.0.smithi049.stdout:verified 2023-10-17T07:17:12.934 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.934 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.935 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.935 INFO:tasks.workunit.client.0.smithi049.stdout:created completion 2023-10-17T07:17:12.935 INFO:tasks.workunit.client.0.smithi049.stdout:started writesame 2023-10-17T07:17:12.935 INFO:tasks.workunit.client.0.smithi049.stdout:expected fail, finished writesame 2023-10-17T07:17:12.935 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:17:12.935 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:17:12.936 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 512 2023-10-17T07:17:12.936 INFO:tasks.workunit.client.0.smithi049.stdout:read completion cb called! 2023-10-17T07:17:12.936 INFO:tasks.workunit.client.0.smithi049.stdout:wrote: 4 2023-10-17T07:17:12.936 INFO:tasks.workunit.client.0.smithi049.stdout:read: 4 2023-10-17T07:17:12.936 INFO:tasks.workunit.client.0.smithi049.stdout:read: 4 2023-10-17T07:17:12.936 INFO:tasks.workunit.client.0.smithi049.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (31951 ms) 2023-10-17T07:17:12.937 INFO:tasks.workunit.client.0.smithi049.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2023-10-17T07:17:12.937 INFO:tasks.workunit.client.0.smithi049.stdout:using new format! 2023-10-17T07:17:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~103s 2023-10-17T07:17:19.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~108s 2023-10-17T07:17:25.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~114s 2023-10-17T07:17:30.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~119s 2023-10-17T07:17:35.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~124s 2023-10-17T07:17:41.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2023-10-17T07:17:46.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2023-10-17T07:17:52.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~141s 2023-10-17T07:17:57.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2023-10-17T07:18:02.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2023-10-17T07:18:08.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-17T07:18:08.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~157s 2023-10-17T07:18:13.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2023-10-17T07:18:13.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~162s 2023-10-17T07:18:18.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2023-10-17T07:18:18.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~167s 2023-10-17T07:18:24.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2023-10-17T07:18:24.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~173s 2023-10-17T07:18:29.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2023-10-17T07:18:29.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~178s 2023-10-17T07:18:34.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2023-10-17T07:18:34.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~183s 2023-10-17T07:18:39.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2023-10-17T07:18:39.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~189s 2023-10-17T07:18:45.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2023-10-17T07:18:45.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~194s 2023-10-17T07:18:50.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2023-10-17T07:18:50.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~199s 2023-10-17T07:18:55.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2023-10-17T07:18:55.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~204s 2023-10-17T07:19:01.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2023-10-17T07:19:01.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~210s 2023-10-17T07:19:02.913 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:02.911+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:06.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2023-10-17T07:19:06.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2023-10-17T07:19:07.914 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:07.913+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:11.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2023-10-17T07:19:11.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2023-10-17T07:19:12.916 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:12.914+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:17.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2023-10-17T07:19:17.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~226s 2023-10-17T07:19:17.917 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:17.916+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:22.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~74s 2023-10-17T07:19:22.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~231s 2023-10-17T07:19:22.918 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:22.918+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:27.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2023-10-17T07:19:27.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~236s 2023-10-17T07:19:27.920 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:27.919+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:32.921 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:32.921+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:33.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2023-10-17T07:19:33.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~242s 2023-10-17T07:19:37.923 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:37.921+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:38.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2023-10-17T07:19:38.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~247s 2023-10-17T07:19:42.924 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:42.922+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:43.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2023-10-17T07:19:43.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~252s 2023-10-17T07:19:47.925 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:47.924+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:48.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2023-10-17T07:19:48.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~257s 2023-10-17T07:19:52.927 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:52.925+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:54.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~106s 2023-10-17T07:19:54.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~263s 2023-10-17T07:19:57.928 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:19:57.927+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:19:59.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~111s 2023-10-17T07:19:59.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~268s 2023-10-17T07:20:02.929 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:02.928+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:04.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2023-10-17T07:20:04.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~273s 2023-10-17T07:20:07.931 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:07.930+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:10.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2023-10-17T07:20:10.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~279s 2023-10-17T07:20:12.932 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:12.930+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:15.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~127s 2023-10-17T07:20:15.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~284s 2023-10-17T07:20:17.934 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:17.932+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:20.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2023-10-17T07:20:20.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~289s 2023-10-17T07:20:22.935 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:22.933+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:26.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2023-10-17T07:20:26.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~295s 2023-10-17T07:20:27.937 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:27.935+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:31.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2023-10-17T07:20:31.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~300s 2023-10-17T07:20:32.938 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:32.937+0000 7f5c911f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.49:6802/1249808404 for 29sec e98 v98) 2023-10-17T07:20:36.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~148s 2023-10-17T07:20:36.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~305s 2023-10-17T07:20:36.643 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-10-17T07:20:36.945 INFO:tasks.ceph.osd.2:Sent signal 15 2023-10-17T07:20:36.946 INFO:tasks.ceph.mon.a:Sent signal 15 2023-10-17T07:20:36.946 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-10-17T07:20:36.947 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:36.946+0000 7f5c941fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 82983) UID: 0 2023-10-17T07:20:36.947 INFO:tasks.ceph.mon.a.smithi049.stderr:2023-10-17T07:20:36.946+0000 7f5c941fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-17T07:20:36.948 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:20:36.947+0000 7fd7f2ff7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 82985) UID: 0 2023-10-17T07:20:36.948 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-17T07:20:36.947+0000 7fd7f2ff7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-17T10:13:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-17T10:13:22.267 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2023-10-17T10:13:22.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-17T10:13:22.567 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi049 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a83d484198747052d1ed9eeed0a72e20b007e411 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2023-10-17T10:13:22.701 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=99d795a4a33f4913a3537ceca74fbbb1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi049 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a83d484198747052d1ed9eeed0a72e20b007e411 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2023-10-17T10:13:22.704 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-17T10:13:22.720 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-17T10:13:22.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T10:13:22.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T10:13:22.841+0000 7f0504d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T10:15:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-17T10:15:22.773 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T10:15:22.798 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-17T10:15:22.798 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-17T10:15:22.799 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-17T10:15:22.799 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T10:15:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T10:15:22.802 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2023-10-17T10:15:22.813 INFO:tasks.ceph.osd.0:Stopped 2023-10-17T10:15:22.814 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-17T10:15:22.814 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T10:15:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T10:15:22.815 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2023-10-17T10:15:22.826 INFO:tasks.ceph.osd.1:Stopped 2023-10-17T10:15:22.827 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-17T10:15:22.827 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T10:15:22.911 INFO:tasks.ceph.osd.2:Stopped 2023-10-17T10:15:22.912 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-17T10:15:22.912 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-17T10:15:22.913 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T10:15:22.913 INFO:tasks.ceph.mgr.x:Stopped 2023-10-17T10:15:22.914 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-17T10:15:22.914 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-17T10:15:22.915 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T10:15:22.915 INFO:tasks.ceph.mon.a:Stopped 2023-10-17T10:15:22.916 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-17T10:15:22.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(POOL_FULL\)' | head -n 1 2023-10-17T10:15:22.953 INFO:teuthology.orchestra.run.smithi049.stdout:2023-10-17T07:19:05.003709+0000 mon.a (mon.0) 276 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec, mon.a has slow ops (SLOW_OPS) 2023-10-17T10:15:22.955 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-17T10:15:22.955 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T10:15:22.956 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-17T10:15:23.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T10:15:23.074 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-17T10:15:23.158 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T10:15:23.159 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-17T10:15:23.223 INFO:tasks.ceph:Archiving mon data... 2023-10-17T10:15:23.225 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067/data/mon.a.tgz 2023-10-17T10:15:23.225 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-10-17T10:15:23.240 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.3J9BHQvV4p 2023-10-17T10:15:23.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3J9BHQvV4p 2023-10-17T10:15:23.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.3J9BHQvV4p 2023-10-17T10:15:23.467 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.3J9BHQvV4p is 248KB 2023-10-17T10:15:23.532 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.3J9BHQvV4p 2023-10-17T10:15:23.547 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-17T10:15:23.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T10:15:23.617 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-17T10:15:23.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-17T10:15:23.689 INFO:teuthology.orchestra.run.smithi049.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-17T10:15:23.694 INFO:tasks.ceph:Archiving crash dumps... 2023-10-17T10:15:23.696 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067/remote/smithi049/crash 2023-10-17T10:15:23.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-17T10:15:23.763 INFO:tasks.ceph:Compressing logs... 2023-10-17T10:15:23.764 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-17T10:15:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-17T10:15:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85072.log 2023-10-17T10:15:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85369.log 2023-10-17T10:15:23.844 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85159.log 2023-10-17T10:15:23.845 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85587.log 2023-10-17T10:15:23.845 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85199.log 2023-10-17T10:15:23.845 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85369.log: /var/log/ceph/ceph-client.admin.85159.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85072.log.gz 2023-10-17T10:15:23.846 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85369.log.gz 2023-10-17T10:15:23.846 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85587.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85159.log.gz 2023-10-17T10:15:23.846 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84804.log 2023-10-17T10:15:23.846 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83225.log 2023-10-17T10:15:23.847 INFO:teuthology.orchestra.run.smithi049.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.85587.log.gz 2023-10-17T10:15:23.847 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85199.log.gz 2023-10-17T10:15:23.847 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.84804.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-17T10:15:23.848 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84804.log.gz 2023-10-17T10:15:23.848 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.83225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84865.log 2023-10-17T10:15:23.848 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83225.log.gz 2023-10-17T10:15:23.848 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83185.log 2023-10-17T10:15:23.849 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.84865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83450.log 2023-10-17T10:15:23.849 INFO:teuthology.orchestra.run.smithi049.stderr: 87.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-17T10:15:23.849 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84865.log.gz 2023-10-17T10:15:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.83185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85071.log 2023-10-17T10:15:23.850 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83185.log.gz 2023-10-17T10:15:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.83450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90215.log 2023-10-17T10:15:23.850 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83450.log.gz 2023-10-17T10:15:23.851 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85073.log 2023-10-17T10:15:23.851 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85071.log.gz 2023-10-17T10:15:23.851 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-17T10:15:23.851 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.90215.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.90215.log.gz 2023-10-17T10:15:23.852 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85327.log 2023-10-17T10:15:23.852 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85073.log.gz 2023-10-17T10:15:23.852 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85198.log 2023-10-17T10:15:23.853 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85285.log 2023-10-17T10:15:23.853 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85327.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85327.log.gz 2023-10-17T10:15:23.853 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85198.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85198.log.gz -5 2023-10-17T10:15:23.853 INFO:teuthology.orchestra.run.smithi049.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84748.log 2023-10-17T10:15:23.854 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-17T10:15:23.854 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.85285.log: /var/log/ceph/ceph-client.admin.84748.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85285.log.gz 2023-10-17T10:15:23.854 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84949.log 2023-10-17T10:15:23.854 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84748.log.gz 2023-10-17T10:15:23.855 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83421.log 2023-10-17T10:15:23.855 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.84949.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84949.log.gz 2023-10-17T10:15:23.855 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83075.log 2023-10-17T10:15:23.856 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83265.log 2023-10-17T10:15:23.856 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.83421.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83421.log.gz 2023-10-17T10:15:23.856 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.83075.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83075.log.gz 2023-10-17T10:15:23.856 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-17T10:15:23.857 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82987.log 2023-10-17T10:15:23.857 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.83265.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83265.log.gz 2023-10-17T10:15:23.857 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84907.log 2023-10-17T10:15:23.858 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.82987.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-17T10:15:23.858 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82987.log.gz 2023-10-17T10:15:23.858 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.84907.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-17T10:15:23.858 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84907.log.gz 2023-10-17T10:15:23.859 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79737.log 2023-10-17T10:15:23.859 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84991.log 2023-10-17T10:15:23.859 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.tmp-client.admin.79737.log: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79737.log.gz 2023-10-17T10:15:23.859 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.84991.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84991.log.gz 2023-10-17T10:15:23.860 INFO:teuthology.orchestra.run.smithi049.stderr: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-17T10:15:23.890 INFO:teuthology.orchestra.run.smithi049.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-17T10:15:24.091 INFO:teuthology.orchestra.run.smithi049.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-17T10:15:24.567 INFO:teuthology.orchestra.run.smithi049.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-17T10:15:25.471 INFO:teuthology.orchestra.run.smithi049.stderr: 96.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-17T10:15:26.080 INFO:teuthology.orchestra.run.smithi049.stderr: 96.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-17T10:15:26.082 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T10:15:26.082 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m2.263s 2023-10-17T10:15:26.082 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m4.731s 2023-10-17T10:15:26.083 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.217s 2023-10-17T10:15:26.083 INFO:tasks.ceph:Archiving logs... 2023-10-17T10:15:26.084 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067/remote/smithi049/log 2023-10-17T10:15:26.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-17T10:15:26.717 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T10:15:26.726 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-17T10:15:26.737 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T10:15:26.741 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-17T10:15:26.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-17T10:15:26.803 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-17T10:15:26.814 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-17T10:15:26.815 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T10:15:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2023-10-17T10:15:26.835 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T10:15:26.835 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-10-17T10:15:26.836 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +1263us[+1250us] +/- 93ms 2023-10-17T10:15:26.836 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 8 377 3 -897us[ -910us] +/- 57ms 2023-10-17T10:15:26.836 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 9 377 92 +129us[ +115us] +/- 60ms 2023-10-17T10:15:26.836 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T10:15:26.837 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-17T10:15:26.847 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-17T10:15:26.848 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-17T10:15:26.859 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-17T10:15:26.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T10:15:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T10:15:27.014 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-17T10:15:27.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-17T10:15:27.035 INFO:teuthology.task.internal:Duration was 11535.259335 seconds 2023-10-17T10:15:27.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-17T10:15:27.048 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-17T10:15:27.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-17T10:15:27.103 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T10:15:27.517 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-17T10:15:27.517 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T10:15:27.518 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-17T10:15:27.538 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-17T10:15:27.539 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T10:15:27.611 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-17T10:15:27.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-17T10:15:27.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-17T10:15:27.764 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-17T10:15:27.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-17T10:15:27.795 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-17T10:15:27.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-17T10:15:27.863 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2023-10-17T10:15:27.887 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-17T10:15:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T10:15:27.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-17T10:15:27.954 INFO:teuthology.task.internal:Transferring archived files... 2023-10-17T10:15:27.955 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067/remote/smithi049 2023-10-17T10:15:27.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-17T10:15:28.024 INFO:teuthology.task.internal:Removing archive directory... 2023-10-17T10:15:28.025 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-17T10:15:28.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-17T10:15:28.086 INFO:teuthology.task.internal:Not uploading archives. 2023-10-17T10:15:28.086 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-17T10:15:28.097 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-17T10:15:28.098 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-17T10:15:28.128 INFO:teuthology.orchestra.run.smithi049.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 17 10:15 /home/ubuntu/cephtest 2023-10-17T10:15:28.129 INFO:teuthology.orchestra.run.smithi049.stdout: 531945 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 17 07:12 /home/ubuntu/cephtest/mnt.0 2023-10-17T10:15:28.129 INFO:teuthology.orchestra.run.smithi049.stdout: 531946 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 17 07:13 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-17T10:15:28.129 INFO:teuthology.orchestra.run.smithi049.stdout: 545856 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 17 07:13 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-17T10:15:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T10:15:28.131 INFO:teuthology.orchestra.run.smithi049.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-17T10:15:28.131 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a83d484198747052d1ed9eeed0a72e20b007e411/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-17T10:15:28.144 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-17T10:15:28.156 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-17T10:15:28.167 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T10:15:28.168 INFO:teuthology.nuke:Checking targets against current locks 2023-10-17T10:15:28.199 DEBUG:teuthology.nuke:shortname: smithi049 2023-10-17T10:15:28.200 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T10:15:28.223 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-10-17_06:50:25-crimson-rados-wip-crimson-scrub-testing-2023-10-16-distro-default-smithi/7431067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 06:52:06.191836', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkX6Uym4JX933EQyRfA93ZVWZvAPT32MnwBP+A/EbyOtGI3nWlbNM/lVWzrn3EPzD/zsLqup9dsaX5G55qcH1Y='} 2023-10-17T10:15:28.271 INFO:teuthology.orchestra.console.smithi049:Power off 2023-10-17T10:15:28.272 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-17T10:15:28.297 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2023-10-17T10:15:28.298 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T10:15:28.320 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-10-17T10:15:32.322 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T10:15:32.435 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-10-17T10:15:36.438 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T10:15:36.548 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2023-10-17T10:15:36.649 INFO:teuthology.orchestra.console.smithi049:Power off completed 2023-10-17T10:15:36.756 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} duration: 11535.259335041046 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi049 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a83d484198747052d1ed9eeed0a72e20b007e411 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_sjust@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=99d795a4a33f4913a3537ceca74fbbb1 status: fail success: false 2023-10-17T10:15:36.757 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T10:15:36.834 INFO:teuthology.run:FAIL